Matt Cutts confirms AdSense media bot in natural search index
April 18, 2006
Matt Cutts confirmed today that the AdSense mediapartners bot (aka mediabot) is indexing pages for use in the Big Daddy Google index. Both Greg Boser and myself have found evidence of mediabot’s crawls for AdSense ad targeting purposes have ended up being used in the natural Google search results. Shoemoney, who attended the Google sponsored luncheon at PubCon reports:
At the Lunch sponsored by Google today Matt Cutts confirmed the recent rumors about media bot results getting into Big Daddy. Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy. Matt also stated that you will gain zero advantage in search listings however if you are serving different content to MediaBot then to Googlebot then you could be in trouble.
It could definitely be used as a tool to detect when content is being cloaked for either the Google or AdSense bot, particularly since the mediapartners bot has been indexing pages since at least the beginning of February.
It will be interesting to see if other consequences arise for webmasters, such as excluding pages for googlebot via robots.txt that end up being indexed via the mediabot. But very nice to see an official confirmation on this from Matt at Google!
Posted in Google AdSense
Subscribe RSS
April 18th, 2006 at 11:47 pm
Getting into the Google Index
JenSense posted about it a few days ago and now it seems that Matt Cutts confirmed it. The Mediabot from Google, the one used for the AdSense targeting is also being used to add sites to the Google Index, before
April 19th, 2006 at 1:30 am
Mediapartners s
April 19th, 2006 at 5:39 am
Lies … Lies … Lies …
I won’t be surprised if they will say that they are using Analytics data and all other data that is contained in Google Accounts .
Time to do my own stats script .
April 19th, 2006 at 6:48 am
Adsense Robot spidert auch für den Google(tm) Index
Wie das Blog jensense.com berichtete spidert der mediapartners bot auch für den Google Index. Dies dürfte erklären, wieso manche Seiten mit Adsense(tm)-Anzeigen in letzter Zeit schon im Index waren, obwohl der eigentliche Googlebot noch …
April 19th, 2006 at 2:16 pm
I’m glad Matt Cutts confirmed that rumor.
During my last job, the company owned litterally hundreds of test sites, and they were more black than white, we noticed that those with Adsense ads on it were still getting cached and indexed, while the others without adsense were de-indexed….
April 19th, 2006 at 2:42 pm
Jen, you raise a really good question. If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index? We all need to know the answer to this one. Maybe somebody can pose the question to Cutts this week.
April 19th, 2006 at 4:43 pm
Thanks for posting this clarification, Jen. I was hoping to do a post about the crawl cache, but haven’t gotten a chance yet b/c of getting pulled into panels here at WMW. Lemme know if you want to chat more about it though.
Aagh. Someone here keeps asking me questions..
April 19th, 2006 at 9:40 pm
Scott you asked “If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index?”
The answer is yes: the Googlebot does the correct/conservative thing; even if a page is in the crawl cache, it will respect the Googlebot rules in the robots.txt file. So we’ll obey the robot.txt rules for each crawler.
April 20th, 2006 at 12:53 am
OMG,,, this is what I still thinking in my mind that how come google bot visit the page that I have never put the link anywhere,, just for testing adsense ads.
April 20th, 2006 at 2:06 am
I don’t know where that fits in, but for what it’s worth - the mediapartners bot is also the one used for the new “Related Links” pseudo-feature. Here comes a new generation of sites with Adsense in a hidden DIV
April 20th, 2006 at 9:06 am
AdSense Spider is a Googlebot
I\’m noticing a few blog posts recently about Google\’s AdSense spider. Notable posts include
April 20th, 2006 at 12:33 pm
Hello guys,
I’ve been following the talks on various forums, while trying to sort out a solution to my everlasting Granking problems.
The matter is quite urgent and I have made a post on Matt Cutt’s blog at http://www.mattcutts.com/blog/seo-advice-discussing-302-redirects/ (the big message by Pavlos, currently at the end there).
Any help would be greatly appreciated. Sorry for posting this here. I am not sure where else.
Regards,
Pavlos Skoufis
April 20th, 2006 at 12:55 pm
A followup on this discussion… My site has about 2500 pages publicly available with adsense ads on each page. However only 500 pages are indexed in Google.
Also on my target keyword a lot of sites, many without unique content appear and us not even on the top 1000 results. However I suspect it has something to do with my previous post above.
On my other sites (news aggregators), all is indexed perfectly, but there are less than 20 pages in all.
April 21st, 2006 at 3:07 pm
This could be good for most, but I am sure will hurt a bunch of people.
April 22nd, 2006 at 12:17 am
Does this mean that being indexed by Adsense (mediapartners) bot ALWAYS is as good as being indexed by the Mozilla Googlebot? Or only sometimes?
This is important because often, there are pages indexed instantly by Mediapartners bot. And the new Mozilla Googlebot seems to have slowed down on several sites in crawling.
So if Mozilla bot crawls 50 pages and Adsense bot crawls another 50, can I assume that 100 pages have been crawled for the index? Or it works that way sometimes?
May 10th, 2006 at 10:55 pm
Googlebot vs mediapartners bot
???????google??????????????googlebot,?google adsense?????????mediapartners bot,????google????,??????,????????????,????????…