A Brief History of Search Engines: From Archie to AI Answers

The timeline

Milestones in web search, in rough order of appearance.

Year	Milestone	Why it mattered
1990	Archie	Indexed FTP filenames — the first practical way to find a file by name.
1991	Gopher and Veronica	Menu-based internet, searchable by keyword.
1993	Wandex, ALIWEB	The first crawlers for the web proper.
1994	WebCrawler, Lycos, Yahoo	Full-text search arrives; Yahoo offers a hand-curated directory.
1995	AltaVista, Excite	Large-scale crawling and fast indexing; AltaVista becomes the default answer machine.
1996–1997	HotBot, Ask Jeeves	Natural-language questions and portal-era competition.
1998	Google	PageRank and link popularity reframe relevance as authority.
2000s	Paid search, personalisation	Advertising funds unlimited infrastructure; local and personalised results grow.
2009	Bing	Microsoft's search relaunched as a competitor with its own ranking stack.
2008–2013	DuckDuckGo, mobile-first	Privacy-first search, instant answers, and a shift to phone-sized queries.
2013–2019	Hummingbird, RankBrain, BERT	Meaning and intent replace string matching; machine learning reaches ranking.
2021	Brave Search, independent engines	Independent crawling and indexes return as a viable niche.
2023–2026	AI-generated answers, chat search	Assistants answer directly, sometimes above or instead of links.

The age of directories

When the web was small enough to describe, directories worked. Yahoo began as a curated list of sites organised by category, maintained by people, and for a couple of years that human judgement was genuinely competitive with automated crawling. The problem was arithmetic: the web grew faster than any editorial team, and the 1990s ended with directories as a pleasant sidebar to algorithmic search rather than a replacement for it.

The age of link ranking

In 1998 two Stanford students argued that the structure of links told you something a document could not say about itself. Treating a link as a vote, weighted by the importance of the voter, produced dramatically better results for the queries that mattered. It also created the template that search has followed ever since: the engine decides, the site owner reacts, and every new signal spawns a new industry trying to manipulate it.

The link era lasted a decade and a half. It ended gradually rather than suddenly, as engines learned that links could be bought, that content could be manufactured, and that what users did with results was itself the best signal of quality.

The age of intent

From around 2013, engines stopped asking "which documents contain these words" and started asking "what does this person want". Query classification, entity understanding, synonym expansion from knowledge graphs and neural language models became part of ranking. Exact-match keyword tactics lost value; pages that answered a question completely — in any words — gained it.

Two side effects are worth noting. First, ranking became less transparent, because a neural model's decisions cannot be explained as a tidy list of matching terms. Second, the results page itself expanded: direct answers, panels, images, maps and shopping boxes, each one an attempt to satisfy an intent without a click.

The age of answers

The most recent shift moves the answer out of the result list entirely. Conversational search and AI overviews combine retrieval with generation: the engine finds sources and writes a response, citing them. For straightforward questions this is convenient, and for publishers it is disruptive, because being cited is not the same as receiving a visit.

Independent engines have grown in parallel, competing on privacy, on their own indexes, or on honest monetisation. The web did not consolidate into one search box after all; the door closed on the 1990s portal era and reopened as a small field of alternatives.

What never changed

Underneath three decades of changing interfaces, the pipeline is identical to the one described in How Search Engines Work. Every generation of search still crawls, still indexes, still ranks — and still depends on the same bargain: publish something useful, make it reachable, and hope it earns its place. The tools got better; the economics of attention did not change at all.