渡虎谷－Yahoo!的新爬虫 : Slurp

最近，Yahoo!宣布其新的网站爬虫已投入使用，根据David Simpson在Yahoo Search Blog上的说明，该新爬虫仍然沿用Yahoo!爬虫的传统名称，Slurp，但索引速度更快更有效率，与它的前辈相比，它大约可节省25%的访问请求和带宽占用。

　　Yahoo!爬虫的效率对我们来说一直是个困扰，以IT技术点评6月份的情况为例，“Yahoo Slurp的读取数据量为170.31M，与之相对的是Googlebot的57.48M和BaiduSpider 39.26M，明显的投入产出比较差，有点可笑的是，6月Yahoo Slurp读取Robots.txt的次数为3104，平均每天100次，不知道是不是因为Yahoo Slurp从不记录Robots.txt的内容，每读取一次网页就要查看一下Robots.txt。”老实说，究竟为什么Yahoo!的效率低到这个程度我们也无从知晓，也许，是Yahoo!与Yahoo!中文造成的重复访问？(纯属猜测)

　　Anyway，希望新的索引爬虫的上线能够让这种情况多多少少得到些改善，不然，真的要考虑是不是干脆在robots.txt中禁掉了Yahoo! Slurp了。

Yahoo! Slurp的介绍
　　从Yahoo!对其的介绍看，应该还是值得期待的。——虽然目前从我们的Web服务器日志上仍看不出有什么改变。

In addition to crawling the Internet faster, our new crawler is more efficient at visiting websites. As a result, site owners will notice as much as a 25% reduction in the number of requests and bandwidth consumed by the crawler.

Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.

　　同时，Yahoo!还对前段时间引人诟病的Yahoo!爬虫“疯狂”的访问提出解释：

While transitioning to the new crawler over the past few weeks, we had been running both crawlers in tandem. In some cases, this increased the frequency of Yahoo Search requests to websites. Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.

With this change of behavior in the crawler, you may see some shuffling of the pages that are included in the index and some changes in ranking as well.