1,000,000,000,000 equals one trillion, which of course is quite a lot. Well Google commented today that they have this many active URL’s in their index. Back in 2000 they hit the billion mark and the first index in 1998 had only 26 million pages.
How do we find all those pages? We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.
