Debating the size of the Web

John Markoff The New York Times: "How big is the World Wide Web? Many Internet engineers consider that query one of those imponderable philosophical questions, like how many angels can dance on the head of a pin.

But the question about the size of the Web came under intense debate last week after Yahoo announced at an Internet search engine conference in Santa Clara, California, that its search engine index - an accounting of the number of documents that can be located from its databases - had reached 19.2 billion.

Because the number was more than twice as large as the 8.1 billion documents currently reported by Google, Yahoo's fierce competitor and Silicon Valley neighbor, the announcement - actually a brief mention in a Yahoo company Web log - set off a spat.

Google questioned the way its rival was counting. Sergey Brin, one of the two co-founders of Google, suggested that the Yahoo index had been inflated with duplicate entries in such a way as to cut its effectiveness despite its size.

The major commercial search engines use software programs known as Web crawlers to scour the Internet systematically for documents and index them. The indexes themselves are maintained as arcane structures of computer data that permit the search engines to return lists of hundreds of answers in fractions of a second when Web users enter terms like "Britney Spears" or "Iraq and weapons of mass destruction."

On Sunday, researchers at the National Center for Supercomputer Applications tried to shed light on the debate by performing a large number of random searches on both indexes. They ran a random sample of 10,012 queries and concluded that Google, on average, returned 166.9 percent more results than Yahoo. In only 3 percent of the cases did the Yahoo searches return more than Google. The group said the Yahoo index claim was suspicious.

Neither Yahoo nor Google makes public the software algorithms that underlie their collection methods. In fact, those details are closely guarded secrets, which lie near the heart of the heated competition now going on among Google, Yahoo and Microsoft over who can provide the most relevant answers to a user's query."