For past months, Google has been beta testing its new indexing system called Caffeine. Caffeine unlike the old indexing system promises to be fast and provide search results instantly. Unlike the old system, which is built on the layer format, each layer updating at a certain interval, caffeine updates the main index simultaneously. Dividing the index into small groups and updating them as the Google Spiders find new content.
Since, web pages are becoming more complex and are full of rich content, search engines such as Google need to step up their efforts in order to provide instant search results to users that are relevant to their interests. To understand how search engines work, check this video from Google:
From the Google Blog:
Our old index had several layers, some of which were refreshed at a faster rate than others; the main layer would update every couple of weeks. To refresh a layer of the old index, we would analyze the entire web, which meant there was a significant delay between when we found a page and made it available to you.
With Caffeine, we analyze the web in small portions and update our search index on a continuous basis, globally. As we find new pages, or new information on existing pages, we can add these straight to the index. That means you can find fresher information than ever before—no matter when or where it was published.
Caffeine lets us index web pages on an enormous scale. In fact, every second Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.
We’ve built Caffeine with the future in mind. Not only is it fresher, it’s a robust foundation that makes it possible for us to build an even faster and comprehensive search engine that scales with the growth of information online, and delivers even more relevant search results to you. So stay tuned, and look for more improvements in the months to come.
As you can see, Caffeine offers major improvements over the old indexing system. The question is, how will this affect us? Well, if you are into forums or like to search for different content, this indexing system will greatly benefit you. You won’t have to wait for a week to see some thread in a forum about your query. Normally when you make a thread in BBS [Forums] the Google spider will index the thread and it will take some time for the search engine to show your thread in search results. However, with Caffeine it usually takes few hours at best.
I have tested this on Zoklet, not only did it help in getting new traffic it also made searching for rare or fresh content easier. In addition, each new optimization will improve video and image indexing. This should make my life a lot easier and Googling more fun.