« Upcoming conferences I am speaking at | Main | AdSense offers text link referral program »

April 18, 2006

Matt Cutts confirms AdSense media bot in natural search index

Matt Cutts confirmed today that the AdSense mediapartners bot (aka mediabot) is indexing pages for use in the Big Daddy Google index. Both Greg Boser and myself have found evidence of mediabot's crawls for AdSense ad targeting purposes have ended up being used in the natural Google search results. Shoemoney, who attended the Google sponsored luncheon at PubCon reports:

At the Lunch sponsored by Google today Matt Cutts confirmed the recent rumors about media bot results getting into Big Daddy. Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy. Matt also stated that you will gain zero advantage in search listings however if you are serving different content to MediaBot then to Googlebot then you could be in trouble.

It could definitely be used as a tool to detect when content is being cloaked for either the Google or AdSense bot, particularly since the mediapartners bot has been indexing pages since at least the beginning of February.

It will be interesting to see if other consequences arise for webmasters, such as excluding pages for googlebot via robots.txt that end up being indexed via the mediabot. But very nice to see an official confirmation on this from Matt at Google!

Posted by Jenstar at April 18, 2006 08:35 PM

Trackback Pings

TrackBack URL for this entry:
http://www.jensense.com/mt-tb.cgi/269

Listed below are links to weblogs that reference Matt Cutts confirms AdSense media bot in natural search index:

» Getting into the Google Index from Oliver Thylmann - Thoughts
JenSense posted about it a few days ago and now it seems that Matt Cutts confirmed it. The Mediabot from Google, the one used for the AdSense targeting is also being used to add sites to the Google Index, before [Read More]

Tracked on April 18, 2006 11:47 PM

» Mediapartners sí indexa para Google from Davilac.net
Hace un par de días, salía la noticia de la sospecha de que el robot Mediapartners de Google estaba indexando para Google. Hoy Matt Cuttse, portavoz oficioso de Google con los SEOs ha confirmado la noticia. Según Cutts, Mediapartners está indexando... [Read More]

Tracked on April 19, 2006 01:30 AM

» Adsense Robot spidert auch für den Google(tm) Index from Webmaster, Internet und SEO Blog
Wie das Blog jensense.com berichtete spidert der mediapartners bot auch für den Google Index. Dies dürfte erklären, wieso manche Seiten mit Adsense(tm)-Anzeigen in letzter Zeit schon im Index waren, obwohl der eigentliche Googlebot noch ... [Read More]

Tracked on April 19, 2006 06:48 AM

» AdSense Spider is a Googlebot from Apogee Weblog
I\'m noticing a few blog posts recently about Google\'s AdSense spider. Notable posts include [Read More]

Tracked on April 20, 2006 09:06 AM

» Avere gli AdSense sul sito non aiuta a farlo indicizzare meglio da Google. O forse si?? from Salatti.NET
GoogleBot e MediaBot sono i due spider di Google che si occupano rispettivamente di indicizzare le pagine per la ricerca e peer contestualizzare gli AdSense. Almeno così volevano far credere Matt Cutts e gli altri ragazzi di Google. A smascherare la c... [Read More]

Tracked on April 30, 2006 02:44 PM

» Googlebot vs mediapartners bot from 關鍵字廣告研究
應該很多人知道google搜尋引擎派來抓網頁的程式叫做googlebot,而google adsense派來分析網頁內容的mediapartners bot,雖然都是google放出來的,不過用途不同,前一個會自動搜尋所有網頁,並且抓到的網頁會... [Read More]

Tracked on May 10, 2006 10:55 PM

Comments

Lies ... Lies ... Lies ...

I won't be surprised if they will say that they are using Analytics data and all other data that is contained in Google Accounts .

Time to do my own stats script .

Posted by: Eugenius at April 19, 2006 05:39 AM

I'm glad Matt Cutts confirmed that rumor.

During my last job, the company owned litterally hundreds of test sites, and they were more black than white, we noticed that those with Adsense ads on it were still getting cached and indexed, while the others without adsense were de-indexed....

Posted by: Nadir at April 19, 2006 02:16 PM

Jen, you raise a really good question. If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index? We all need to know the answer to this one. Maybe somebody can pose the question to Cutts this week.

Posted by: Scott Johnson at April 19, 2006 02:42 PM

Thanks for posting this clarification, Jen. I was hoping to do a post about the crawl cache, but haven't gotten a chance yet b/c of getting pulled into panels here at WMW. Lemme know if you want to chat more about it though.

Aagh. Someone here keeps asking me questions.. :)

Posted by: Matt Cutts at April 19, 2006 04:43 PM

Scott you asked "If a page is crawled via MediaBot, does Google respect the Googlebot rules in robots.txt when it considers storing the page in the index?"

The answer is yes: the Googlebot does the correct/conservative thing; even if a page is in the crawl cache, it will respect the Googlebot rules in the robots.txt file. So we'll obey the robot.txt rules for each crawler.

Posted by: Matt Cutts at April 19, 2006 09:40 PM

OMG,,, this is what I still thinking in my mind that how come google bot visit the page that I have never put the link anywhere,, just for testing adsense ads.

Posted by: Tony at April 20, 2006 12:53 AM

I don't know where that fits in, but for what it's worth - the mediapartners bot is also the one used for the new "Related Links" pseudo-feature. Here comes a new generation of sites with Adsense in a hidden DIV :-)

Posted by: John at April 20, 2006 02:06 AM

Hello guys,

I've been following the talks on various forums, while trying to sort out a solution to my everlasting Granking problems.

The matter is quite urgent and I have made a post on Matt Cutt's blog at http://www.mattcutts.com/blog/seo-advice-discussing-302-redirects/ (the big message by Pavlos, currently at the end there).

Any help would be greatly appreciated. Sorry for posting this here. I am not sure where else.

Regards,
Pavlos Skoufis

Posted by: Pavlos at April 20, 2006 12:33 PM

A followup on this discussion... My site has about 2500 pages publicly available with adsense ads on each page. However only 500 pages are indexed in Google.

Also on my target keyword a lot of sites, many without unique content appear and us not even on the top 1000 results. However I suspect it has something to do with my previous post above.

On my other sites (news aggregators), all is indexed perfectly, but there are less than 20 pages in all.

Posted by: Pavlos at April 20, 2006 12:55 PM

This could be good for most, but I am sure will hurt a bunch of people.

Posted by: Jake at April 21, 2006 03:07 PM

Does this mean that being indexed by Adsense (mediapartners) bot ALWAYS is as good as being indexed by the Mozilla Googlebot? Or only sometimes?

This is important because often, there are pages indexed instantly by Mediapartners bot. And the new Mozilla Googlebot seems to have slowed down on several sites in crawling.

So if Mozilla bot crawls 50 pages and Adsense bot crawls another 50, can I assume that 100 pages have been crawled for the index? Or it works that way sometimes?

Posted by: Matt at April 22, 2006 12:17 AM