« Site Diagnostics tab returns to Google AdSense control panel | Main | AdSense testing new style of vertical images in ad units »

July 21, 2006

Google confirms use of AdSense bot for regular search results cache

While Matt Cutts confirmed the use of the AdSense mediapartners bot for the natural search results cache at both WebmasterWorld Pubcon and on his blog, there still had not been anything official about it on the Google website itself (although Matt is just about as official as you can get without it being on Google.com!) But with some new additions to the Google AdSense help pages to coincide with the new Site Diagnostics tab, the link has now been officially confirmed.

From the "How do your crawler and site diagnostic reports work?" support page:

The AdSense crawler is different from the Google crawler

The two crawlers are separate, but they do share a cache. We do this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Sitemaps crawler is separate.

There still doesn't seem to be any mention of this in the regular Google webmaster guidelines, although this could be updated in the future.

Posted by Jenstar at July 21, 2006 11:26 AM

Comments

The million dollar question for me is, "If I replace Adsense on my site with YPN, will my site be crawled less frequently?" Officially, Google will say it doesn't matter. However, my understanding is that when Adsense encounters a new page it sends out the mediapartners bot to crawl the new page. So, if you no longer run Adsense, the crawling will be done by Googlebot, which has its own schedule for crawling your site.

Posted by: Ken at July 21, 2006 01:47 PM

Hi Ken, let me make your question more specific: "If I replace Adsense on my site with YPN, will my site have fewer pages in Google's index?" and the answer is no. You'll have the same number of pages in the index either way. To answer your original question, you'd be crawled less frequently for AdSense purposes of course (since you left AdSense), but you'd be crawled exactly the same for Googlebot/indexing purposes. See my blog post at
http://www.mattcutts.com/blog/crawl-caching-proxy/
if you want a picture and nitty gritty details.

Also, Jen had a post here:
http://www.jensense.com/archives/2006/04/adsense_mediapa.html
where she pointed out some stale/incorrect documentation at this url:
https://www.google.com/support/adsense/bin/answer.py?answer=9717

Thanks to Jen for noticing that; we updated that answer to be more accurate. Nonetheless, I'm glad that there's an answer on the Google site that explicitly mentions the cache now.

Posted by: Matt Cutts at July 22, 2006 02:30 PM

Matt, thanks for taking the time (on a Saturday even) to help me understand the crawl caching.

Posted by: Ken at July 23, 2006 11:51 AM