Back To Other Google Information and Resources

Robots.txt

  • Dubya
  • Proficient
  • Proficient
  • User avatar
  • Joined: Oct 22, 2004
  • Posts: 468
  • Status: Offline

Post November 15th, 2004, 10:50 pm

Ok, I've heard good and bad things about having a robots.txt file on your server. Is there a benefit to having this file, or does it hinder the google bots from searching your site. Does anyone know?
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post November 15th, 2004, 10:50 pm

  • Maedhros
  • Proficient
  • Proficient
  • User avatar
  • Joined: Oct 31, 2004
  • Posts: 325
  • Loc: Durham, England
  • Status: Offline

Post November 16th, 2004, 6:47 am

It should only hinder robots if you tell it to... and it could potentially be bad if you don't have one. For instance, if your server automatically redirects you to another (html) page on a 404 error, the robot will get that as well when it looks for the robot.txt file, which can't really be good...
Gentoo Linux: "All of a sudden, Larry the Cow was in control. And he liked it."
  • Dubya
  • Proficient
  • Proficient
  • User avatar
  • Joined: Oct 22, 2004
  • Posts: 468
  • Status: Offline

Post November 16th, 2004, 9:58 pm

The forums part of my website has a robots.txt file (it's phpBB), but the main part of my website does not have one. You think I should add one?
  • Jess
  • Guru
  • Guru
  • User avatar
  • Joined: Sep 10, 2004
  • Posts: 1153
  • Loc: USA
  • Status: Offline

Post November 16th, 2004, 10:01 pm

the first thing a search spider looks for when visiting your site is the robots.txt file. It's worth having one even if its just to tell the spider that it can access all your files (if you dont want to block anything)
  • smo
  • Beginner
  • Beginner
  • No Avatar
  • Joined: Nov 06, 2004
  • Posts: 43
  • Loc: India
  • Status: Offline

Post November 16th, 2004, 11:54 pm

Dubya wrote:
The forums part of my website has a robots.txt file (it's phpBB), but the main part of my website does not have one. You think I should add one?

It should be in your root. That is where the the robots will look for.
  • webcertain
  • Beginner
  • Beginner
  • No Avatar
  • Joined: Oct 28, 2003
  • Posts: 46
  • Loc: uk
  • Status: Offline

Post November 17th, 2004, 3:59 am

May I recommend this great tutorial from Ian McAnerin? It's a great guide to robots.txt
Cheers,
Johann

Page 1 of 1

Post Information

  • Total Posts in this topic: 6 posts
  • Users browsing this forum: No registered users and 31 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.