what robots do I need to block?

  • tastysite
  • Proficient
  • Proficient
  • User avatar
  • Posts: 349
  • Loc: Brighouse, West Yorkshire, England

Post 3+ Months Ago

What robots should I block using robots.txt ie ones that hit a lot of times or ones that cores problems etc?
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • onlyway
  • Newbie
  • Newbie
  • User avatar
  • Posts: 5
  • Loc: www.onlywaytopark.com

Post 3+ Months Ago

Google, Yahoo, Cuil, Jayde, MSN, Yesup are the one that can create problems and eat up bandwidth.

The best thing you can do is to add "index, nofollow" on your pages if you want them to be indexed without problems. "nofollow" will tell the bots not to follow the hyperlinks on your website, thus saving you bandwidth and problems.
Moderator Remark: Removed manual signature

Post Information

  • Total Posts in this topic: 2 posts
  • Users browsing this forum: No registered users and 5 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.