Jenstar

Sponsors

Matt Cutts confirms AdSense media bot in natural search index

Matt Cutts confirmed today that the AdSense mediapartners bot (aka mediabot) is indexing pages for use in the Big Daddy Google index. Both Greg Boser and myself have found evidence of mediabot’s crawls for AdSense ad targeting purposes have ended up being used in the natural Google search results. Shoemoney, who attended the Google sponsored luncheon at PubCon reports:

At the Lunch sponsored by Google today Matt Cutts confirmed the recent rumors about media bot results getting into Big Daddy. Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy. Matt also stated that you will gain zero advantage in search listings however if you are serving different content to MediaBot then to Googlebot then you could be in trouble.

It could definitely be used as a tool to detect when content is being cloaked for either the Google or AdSense bot, particularly since the mediapartners bot has been indexing pages since at least the beginning of February.

It will be interesting to see if other consequences arise for webmasters, such as excluding pages for googlebot via robots.txt that end up being indexed via the mediabot. But very nice to see an official confirmation on this from Matt at Google!

Share this with others!
  • Twitter
  • Digg
  • Sphinn
  • StumbleUpon
  • del.icio.us
  • Reddit
  • Technorati
  • Mixx
  • Google Bookmarks
  • Facebook

16 comments to Matt Cutts confirms AdSense media bot in natural search index

  • Getting into the Google Index

    JenSense posted about it a few days ago and now it seems that Matt Cutts confirmed it. The Mediabot from Google, the one used for the AdSense targeting is also being used to add sites to the Google Index, before

  • Lies … Lies … Lies …

    I won’t be surprised if they will say that they are using Analytics data and all other data that is contained in Google Accounts .

    Time to do my own stats script .

  • Adsense Robot spidert auch für den Google(tm) Index

    Wie das Blog jensense.com berichtete spidert der mediapartners bot auch für den Google Index. Dies dürfte erklären, wieso manche Seiten mit Adsense(tm)-Anzeigen in letzter Zeit schon im Index waren, obwohl der eigentliche Googlebot noch …

  • I’m glad Matt Cutts confirmed that rumor.

    During my last job, the company owned litterally hundreds of test sites, and they were more black than white, we noticed that those with Adsense ads on it were still getting cached and indexed, while the others without adsense were de-indexed….

  • Jen, you raise a really good question. If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index? We all need to know the answer to this one. Maybe somebody can pose the question to Cutts this week.

  • Thanks for posting this clarification, Jen. I was hoping to do a post about the crawl cache, but haven’t gotten a chance yet b/c of getting pulled into panels here at WMW. Lemme know if you want to chat more about it though.

    Aagh. Someone here keeps asking me questions.. :)

  • Scott you asked “If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index?”

    The answer is yes: the Googlebot does the correct/conservative thing; even if a page is in the crawl cache, it will respect the Googlebot rules in the robots.txt file. So we’ll obey the robot.txt rules for each crawler.

  • OMG,,, this is what I still thinking in my mind that how come google bot visit the page that I have never put the link anywhere,, just for testing adsense ads.

  • I don’t know where that fits in, but for what it’s worth – the mediapartners bot is also the one used for the new “Related Links” pseudo-feature. Here comes a new generation of sites with Adsense in a hidden DIV :-)

  • AdSense Spider is a Googlebot

    I\’m noticing a few blog posts recently about Google\’s AdSense spider. Notable posts include

  • Pavlos

    Hello guys,

    I’ve been following the talks on various forums, while trying to sort out a solution to my everlasting Granking problems.

    The matter is quite urgent and I have made a post on Matt Cutt’s blog at http://www.mattcutts.com/blog/seo-advice-discussing-302-redirects/ (the big message by Pavlos, currently at the end there).

    Any help would be greatly appreciated. Sorry for posting this here. I am not sure where else.

    Regards,
    Pavlos Skoufis

  • Pavlos

    A followup on this discussion… My site has about 2500 pages publicly available with adsense ads on each page. However only 500 pages are indexed in Google.

    Also on my target keyword a lot of sites, many without unique content appear and us not even on the top 1000 results. However I suspect it has something to do with my previous post above.

    On my other sites (news aggregators), all is indexed perfectly, but there are less than 20 pages in all.

  • This could be good for most, but I am sure will hurt a bunch of people.

  • Does this mean that being indexed by Adsense (mediapartners) bot ALWAYS is as good as being indexed by the Mozilla Googlebot? Or only sometimes?

    This is important because often, there are pages indexed instantly by Mediapartners bot. And the new Mozilla Googlebot seems to have slowed down on several sites in crawling.

    So if Mozilla bot crawls 50 pages and Adsense bot crawls another 50, can I assume that 100 pages have been crawled for the index? Or it works that way sometimes?

  • Googlebot vs mediapartners bot

    ???????google??????????????googlebot,?google adsense?????????mediapartners bot,????google????,??????,????????????,????????…