How Search Engines Make Sense of the Web

By Chris Sherman, Associate Editor Search Engine Watch
May 5, 2003

Search engines are essentially massive full-text indexes of web pages. The quality of the indexes, and how the engines use the information they contain, is what makes -- or breaks -- the quality of search results.

We're all familiar with back-of-the-book indexes. They're simply alphabetized lists of the important words in the book, and the pages on which they appear.

Search engine indexes are similar, but vastly more complex that back-of-the-book indexes. Although most of us will never want to become experts on web indexing, knowing even a little bit about how they're built and used can vastly improve your searching skills.

A good way to learn about web indexing is to spend some time with a page compiled by The School of Library, Archival and Information Studies at the University of British Columbia. Indexing Resources on the WWW is a lengthy list of links to research, articles and web sites concerned with the process of indexing of all kinds.

SearchDay Readers will want to investigate two sections in particular. The first, Information Retrieval, looks at all types of searching.

Although most of the links point to technical information, there are a number of excellent articles for non-professionals, such as Vannevar Bush's classic As We May Think, the recent U.C. Berkeley research on How Much Information, and others.

The second section, Indexing Resources on the WWW is focused on indexing specifically for the world wide web. Three parts of this page will be of most interest:

WWW Indexing
Web Site Indexes
Automatic Indexing for the WWW