How to keep Yahoo from crawling one of my sites.

  • roma
  • Graduate
  • Graduate
  • roma
  • Posts: 142

Post 3+ Months Ago

I want to stop Yahoo from crawling a site. I've read it doesn't obey the robots.txt file. What string do I use in .htaccess and/or robots.txt to accomplish this?

Thank you.

sk
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Posts: 23456
  • Loc: Woodbridge VA

Post 3+ Months Ago

Well, it definitely "reads" the robots.txt file The identifier is Inktomi Slurp but I have also read where it is also now known as Yahoo! Slurp (although I've only seen Inktomi Slurp in my log. Add this to your robots.txt file and monitor the results to see if it works.

User-Agent: Inktomi Slurp
Disallow /

User-Agent: Yahoo! Slurp
Disallow /

Not sure why you'd want to do that other than to conserve bandwidth, though.
  • roma
  • Graduate
  • Graduate
  • roma
  • Posts: 142

Post 3+ Months Ago

The reason I'm doing this --- and I'm not sure it's a good idea or not --- is that I have two sites that promote the same service. They're on different IPs and servers and look different. But there is some common text, though not identical. I'm listed in Yahoo for my old site and don't want that botched up. So I thought I should prohibit Yahoo and MSN from crawling the new site since I am indexed with them. I'm banned from Google with the old site but not the new. So I'm not sure what to do there.

Any comments on taking this sort of action?

Post Information

  • Total Posts in this topic: 3 posts
  • Users browsing this forum: No registered users and 2 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.