Googlebot requests robot.txt over 100 times a day

[zthomasz]

According to google the last time googlebot accessed my homepage was Feb 18, 07 ... five months ago. I checked cache:www.rightline.net today and goog doesn't even have the homepage cached.

Our site is 10 years old, and has always ranked high for a large number of important key words. Now some of our top pages aren't even indexed.

Here is an example (from the logs) of googlebot's excessive requests for robot.txt ...

66.249.66.82 - - [02/Jul/2007:12:52:17 -0700] "GET /robots.txt HTTP/1.1" 200 25 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Googlebot has made this same request over 120 times today (Mon Jul 2 07) and 1,000's of times since mid-May.

I thought it might be an apache problem (linux) so I've had several IT guys working on this over the weekend. As of now, they're all stumped.

Thanks,

-Thomas


PS - From google.com/support/webmasters/ ... "To save bandwidth, Googlebot only downloads the robots.txt file once a day or whenever we've fetched many pages from the server."