Jenstar

Sponsors

Google confirms use of AdSense bot for regular search results cache

While Matt Cutts confirmed the use of the AdSense mediapartners bot for the natural search results cache at both WebmasterWorld Pubcon and on his blog, there still had not been anything official about it on the Google website itself (although Matt is just about as official as you can get without it being on Google.com!) But with some new additions to the Google AdSense help pages to coincide with the new Site Diagnostics tab, the link has now been officially confirmed.

From the “How do your crawler and site diagnostic reports work?” support page:

The AdSense crawler is different from the Google crawler

The two crawlers are separate, but they do share a cache. We do this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Sitemaps crawler is separate.

There still doesn’t seem to be any mention of this in the regular Google webmaster guidelines, although this could be updated in the future.

Share this with others!
  • Twitter
  • Digg
  • Sphinn
  • StumbleUpon
  • del.icio.us
  • Reddit
  • Technorati
  • Mixx
  • Google Bookmarks
  • Facebook

3 comments to Google confirms use of AdSense bot for regular search results cache

  • Ken

    The million dollar question for me is, “If I replace Adsense on my site with YPN, will my site be crawled less frequently?” Officially, Google will say it doesn’t matter. However, my understanding is that when Adsense encounters a new page it sends out the mediapartners bot to crawl the new page. So, if you no longer run Adsense, the crawling will be done by Googlebot, which has its own schedule for crawling your site.

  • Hi Ken, let me make your question more specific: “If I replace Adsense on my site with YPN, will my site have fewer pages in Google’s index?” and the answer is no. You’ll have the same number of pages in the index either way. To answer your original question, you’d be crawled less frequently for AdSense purposes of course (since you left AdSense), but you’d be crawled exactly the same for Googlebot/indexing purposes. See my blog post at
    http://www.mattcutts.com/blog/crawl-caching-proxy/
    if you want a picture and nitty gritty details.

    Also, Jen had a post here:
    http://www.jensense.com/archives/2006/04/adsense_mediapa.html
    where she pointed out some stale/incorrect documentation at this url:
    https://www.google.com/support/adsense/bin/answer.py?answer=9717

    Thanks to Jen for noticing that; we updated that answer to be more accurate. Nonetheless, I’m glad that there’s an answer on the Google site that explicitly mentions the cache now.

  • Ken

    Matt, thanks for taking the time (on a Saturday even) to help me understand the crawl caching.