Robots.txt

  • Dubya
  • Proficient
  • Proficient
  • User avatar
  • Posts: 468

Post 3+ Months Ago

Ok, I've heard good and bad things about having a robots.txt file on your server. Is there a benefit to having this file, or does it hinder the google bots from searching your site. Does anyone know?
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • Maedhros
  • Proficient
  • Proficient
  • User avatar
  • Posts: 325
  • Loc: Durham, England

Post 3+ Months Ago

It should only hinder robots if you tell it to... and it could potentially be bad if you don't have one. For instance, if your server automatically redirects you to another (html) page on a 404 error, the robot will get that as well when it looks for the robot.txt file, which can't really be good...
  • Dubya
  • Proficient
  • Proficient
  • User avatar
  • Posts: 468

Post 3+ Months Ago

The forums part of my website has a robots.txt file (it's phpBB), but the main part of my website does not have one. You think I should add one?
  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

the first thing a search spider looks for when visiting your site is the robots.txt file. It's worth having one even if its just to tell the spider that it can access all your files (if you dont want to block anything)
  • smo
  • Beginner
  • Beginner
  • smo
  • Posts: 43
  • Loc: India

Post 3+ Months Ago

Dubya wrote:
The forums part of my website has a robots.txt file (it's phpBB), but the main part of my website does not have one. You think I should add one?

It should be in your root. That is where the the robots will look for.
  • webcertain
  • Beginner
  • Beginner
  • webcertain
  • Posts: 46
  • Loc: uk

Post 3+ Months Ago

May I recommend this great tutorial from Ian McAnerin? It's a great guide to robots.txt
Cheers,
Johann

Post Information

  • Total Posts in this topic: 6 posts
  • Users browsing this forum: No registered users and 3 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.