MSN - Not following the Robots.txt causing me issues

  • ctrenks
  • Student
  • Student
  • ctrenks
  • Posts: 87

Post 3+ Months Ago

My site http://www.allfreechips.com has about 12 copies of my index cached on MSN as I have a referral code i added. So even though robots.txt denies ?ref= MSN still indexes them causing me huge issues with duplicate content I guess. This has been going on since the live move and I wrote to MSN a couple of times. I also have a directory restricted allfreechips.com/cportal/ yet they cache hundreds of pages from there.

any thoughs if im missing somthing.

http://www.allfreechips.com/robots.txt
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Posts: 23456
  • Loc: Woodbridge VA

Post 3+ Months Ago

This will disallow all bots:

Quote:
User-agent: *
Disallow: /phpBB2/admin/
Disallow: /phpBB2/db/
Disallow: /phpBB2/images/
Disallow: /phpBB2/includes/
Disallow: /phpBB2/language/
Disallow: /phpBB2/templates/
Disallow: /phpBB2/common.php
Disallow: /phpBB2/config.php
Disallow: /phpBB2/faq.php
Disallow: /phpBB2/groupcp.php
Disallow: /phpBB2/login.php
Disallow: /phpBB2/memberlist.php
Disallow: /phpBB2/modcp.php
Disallow: /phpBB2/posting.php
Disallow: /phpBB2/privmsg.php
Disallow: /phpBB2/profile.php
Disallow: /phpBB2/search.php
Disallow: /phpBB2/viewonline.php
Disallow: /phpBB2/-vp.html$
Disallow: /cportal/
Disallow: /c/
Disallow: /?ref=
Disallow: /?sort=
Disallow: /afc-rewards/
Disallow: /casino-software/


You need the Asterick "wildcard" to flag all bots. - that would include MSN. Right now you are only denying Google and Inktomi. That middle section isn't doing anything because you don't have a user agent specified.
  • ctrenks
  • Student
  • Student
  • ctrenks
  • Posts: 87

Post 3+ Months Ago

Woo Hoo.. Thanks for the feedback on this one, lets see what happens :)
  • factoring
  • Novice
  • Novice
  • factoring
  • Posts: 20

Post 3+ Months Ago

You could also try putting a robots message in your head tag.

Post Information

  • Total Posts in this topic: 4 posts
  • Users browsing this forum: No registered users and 1 guest
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.