Matt Cutts confirmed today that the AdSense mediapartners bot (aka mediabot) is indexing pages for use in the Big Daddy Google index. Both Greg Boser and myself have found evidence of mediabot’s crawls for AdSense ad targeting purposes have ended up being used in the natural Google search results. Shoemoney, who attended the Google sponsored luncheon at PubCon reports:
At the Lunch sponsored by Google today Matt Cutts confirmed the recent rumors about media bot results getting into Big Daddy. Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy. Matt also stated that you will gain zero advantage in search listings however if you are serving different content to MediaBot then to Googlebot then you could be in trouble.
It could definitely be used as a tool to detect when content is being cloaked for either the Google or AdSense bot, particularly since the mediapartners bot has been indexing pages since at least the beginning of February.
It will be interesting to see if other consequences arise for webmasters, such as excluding pages for googlebot via robots.txt that end up being indexed via the mediabot. But very nice to see an official confirmation on this from Matt at Google!