« A look at the new Content Referral CPA from Google AdSense | Main | New AdSense "Vertical Images" within ad units »

June 24, 2006

Yahoo Search Marketing Bot goes wild with 83,000 visits to single page in 24 hours

As many publishers know, the Yahoo Search Marketing bot - also known by the "YahooYSMcm/2.0.0" user agent - is responsible for visiting publisher pages in order to provide targeted ads. However, after investigating a massive spike in my bandwidth for JenSense from June 14th and 15th, I discovered that the YSM bot apparently really liked my blog entry from last summer comparing the Yahoo Publisher Network and Google AdSense. So much so that it decided to visit 83,000 times over a 24 hour period (well, 23 hours and 38 minutes, to be exact).

Yes, that is right, the YSM bot came as often as three times a second, but mostly every second. The longest break it seemed to take once it hit its groove was simply leaving a single second between visits. And it made for a grand total of 82985 visits by this hungry bot. And that makes for a nasty 3 GB of bandwidth that this bot used just on this single page.

It first showed up at 6:01 am PST on June 14th and the non-stop visits continued through to 5:39 am PST the following day.

It is worth noting that the visits came from three separate IPs - 63.163.102.180, 63.163.102.181 and 63.163.102.182, meaning that it wasn't a single bot having a bad day, but it was three of them. And those bots do trace back to Yahoo Search Marketing (as Overture Services).

Here is a bit of the raw logs a minute or so before it stopped (logs are EST):

63.163.102.180 - - [15/Jun/2006:08:38:40 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:41 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:41 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.182 - - [15/Jun/2006:08:38:42 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:43 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.182 - - [15/Jun/2006:08:38:45 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:45 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.182 - - [15/Jun/2006:08:38:46 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:47 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:47 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:48 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:50 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.182 - - [15/Jun/2006:08:38:51 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:52 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:52 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:53 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:54 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:56 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:56 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:57 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:58 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.181 - - [15/Jun/2006:08:38:58 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

63.163.102.180 - - [15/Jun/2006:08:38:59 -0400] "GET /archives/2005/08/womensfinanceco.html HTTP/1.1" 200 33712 "-" "YahooYSMcm/2.0.0"

I do have to wonder how no one from Yahoo could have noticed that the bots were going crazy on this particular page, especially on a page that generally only has a handful of page views per day. I never heard anything about this from Yahoo, I only discovered it when investigating the source of the massive traffic spike.

However, it is possible they noticed it, realized whose site they did it to, said a few colorful adjectives, and then crossed their fingers and hoped I wouldn't notice ;) If I were in their shoes, I would probably do the same!

However, this problem could definitely affect publishers who might not have the money or hosting arrangement to allow for the increased bandwidth the YSM bot used. Hopefully this was an isolated incident and I was the (un)lucky one to be on the receiving end of it.

Posted by Jenstar at June 24, 2006 12:16 AM

Trackback Pings

TrackBack URL for this entry:
http://www.jensense.com/mt-tb.cgi/300

Comments

I've blocked or throttled various Yahoo bots on my sites because of incidents like this that have happened to me. Not nice.

Posted by: Brad at June 24, 2006 08:26 AM

Another reason that I am glad when YPN gave me a hard time over the phone back in its early days, I dumped them. Would my few tiny blogs have gained a few cents had I submitted to their rudeness? Perhaps, but I'm currently YPN free and glad of it. Google treats me as a publisher/partner, not a doormat.

Posted by: Dave Starr at June 24, 2006 11:59 AM

Post a comment




Remember Me?