In 1994, Brian Pinkerton, a Computer Science and Engineering student at the University of Washington, used his spare time to create Web Crawler.
With Web Crawler, Brian generated a list of the Top 25 websites on March 15, 1994.
A little less than a year later, Web Crawler was fully operating on advertising revenue.
A young America Online, without access to the web, acquired Web Crawler on June 1, 1995.
While both web directories and search engines gained popularity in the 1990s, search engines developed a life of their own becoming the preferred method of Internet search.
Toward the end of the 1990s, Excite partnered with MSN and Netscape providing search services.
Info Space’s Dogpile crawler replaced Excite’s making Dogpile and Excite’s search results the same.
Both Excite and Dogpile are also powered by Look Smart’s directory, except that Dogpile includes a number of other Info Space directories.
In October 2002, Yahoo shifted to crawler-based listing for its search results. purchased Overture’s pay-per-click service, which had only months earlier purchased Alta Vista and Allthe Web, and Inktomi’s search database. combined these tools to create its own search index. Search Marketing and provides paid search advertising revenue. Directory remains one of the top indexes powering search listings.
Web Crawler was the first search engine to provide full text search.