Back To Other Google Information and Resources

Bandwidth problem due to bots

  • Molon Lave
  • Born
  • Born
  • No Avatar
  • Joined: Apr 10, 2008
  • Posts: 2
  • Status: Offline

Post April 10th, 2008, 12:04 pm

Hello guys, I'm not sure if this is the correct section to post this problem in. Mods, please move to the appropriate section if this is the wrong place to be in.

Ok, here it is, ever since I have changed from 2.1.7 to 2.3.3 IPB boards, my bandwidth has gone thru the roof. I used to go thru about 30 gigs a month, know I go thru 120 gig, and the google bot eats up about 85 gig, the yahoo bot another 10 gig.

My limit is 40 gig and each gig after that is $4 a pop.

Now the problem, I rank very high on each search engine, and dont want to lose it, but I need to control the amount of bandwidth the bots go thru. I dont want to ban them altogether either as i will lose my search engine ranking.

I found this in regards to the yahoo bot
http://help.yahoo.com/l/us/yahoo/search ... rp-03.html

Quote:
You can add a "Crawl-delay: x.x" instruction, where "x.x" is a delay value between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 5 or 10 or a comfortable value for your server, but we suggest you start with small values (0.5–1), and increase as needed to an acceptable value for your server. Larger delay values add more delay between successive crawl accesses and decrease the maximum crawl rate to your web server.

For example, a robots.txt rule to set a crawl-delay of 5 for Yahoo! Slurp looks like:

User-agent: Slurp

Crawl-delay: 5

A shorter delay value of 0.5 would look like:

User-agent: Slurp

Crawl-delay: 0.5


Now, I have a VPS server for my site, so my questions are as follows.

What value should I use, and is there is a similar thing I can do for a google bot?

Or is there another way to reduce the amount of bandwidth the bots consume?
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post April 10th, 2008, 12:04 pm

  • joebert
  • Sledgehammer
  • Genius
  • No Avatar
  • Joined: Feb 10, 2004
  • Posts: 13458
  • Loc: Florida
  • Status: Offline

Post April 10th, 2008, 10:32 pm

http://www.google.com/support/webmaster ... swer=48620
Strong with this one, the sudo is.
  • Molon Lave
  • Born
  • Born
  • No Avatar
  • Joined: Apr 10, 2008
  • Posts: 2
  • Status: Offline

Post April 11th, 2008, 8:26 am

joebert wrote:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620



Thank you so much for your help. I'll give it a try and see how I go. Will reply here as to the outcome.

Page 1 of 1

Post Information

  • Total Posts in this topic: 3 posts
  • Users browsing this forum: No registered users and 19 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.