Has Google Reached It’s Limit?
By Dylan Downhill
Google went public last month but that’s not the big news for Google. Nor is the settlement of the PPC patent fight with Overture. The big news is that Google may have hit the limit on the number of pages it can store. In a nutshell the number of pages Google says it has indexed (currently it reads ‘Searching 4,285,199,774 web pages’ is not that far from the largest number an integer in Unix//Linux can handle – around 10 million off – see full article here http://www.w3reports.com/index.php?itemid=549. Whether they really can’t add new pages without deleting old ones or whether this is bologna (after all, they are a bunch of uber-techies – surely they increased the size of the index when they saw this coming); only Google knows.
Why did I mention this. Well we’ve been noticing a definite lag time for new sites to get fully indexed. Older sites are still being indexed fine, and we’ve noticed that adding a new page with a link from the home page will get it indexed extremely quickly (although the Page Rank takes a long time to catch up). We have noticed with older sites that rename the URL of hundreds of pages all at once (such as during a redesign) are not being indexed quickly either.
We have noticed that Yahoo search results have been getting better, and with the lag that Google has introduced we find ourselves heading to Yahoo more often than we used to, and with the toning down of the advertising the overall experience at Yahoo is more pleasing than it was (take note Ask Jeeves!)
In terms of results we have noticed that Ask Jeeves/Teoma will show ranking improvements quickly, then Yahoo follows soon afterwards. Google is taking a long time to respond to site changes.
To ensure your site gets fully indexed we do recommend that you add a good sitemap linking to all the pages you want googlebot (the Google spider) to find. In fact the process of building a sitemap can help you find orphaned pages (no longer linked to from anywhere on the site), I added one recently to Elixir and found 2 pages that were orphaned.