"The search engines seem to be showing signs of strain in attempting to keep up with the explosive growth of the Web. Steve Lawrence & C. Lee Giles of the NEC Research Center conducted a scientifically rigorous survey of the search engine’s coverage of Web content in February 1999.

The findings of their survey, published in the peer-reviewed journal Nature, suggest that the combined coverage of the 11 search engines used for the study was about 42% of the total number of unique indexable pages on the Web (i.e. not including the ever-expanding "hidden Web"), with no search engine indexing more than about 16%."

From Gill, Tony. Metadata and the World Wide Web (2000)
http://www.getty.edu/research/conducting_research/standards/intrometadata/2_articles/gill/content.html#introduction

Introduction to Metadata V.2.0
http://www.getty.edu/research/conducting_research/standards/intrometadata/1_introduction/index.html