Back To Other Search Engines & Directories

robots.txt

  • KozSEO
  • Newbie
  • Newbie
  • No Avatar
  • Joined: Nov 19, 2004
  • Posts: 8
  • Loc: USA
  • Status: Offline

Post November 19th, 2004, 9:48 am

My log files keep showing 404 errors on my robots.txt file
I had the file validated and it seems fine, here is the file

# Allows all robots

User-agent: *
Disallow:

I have just started using the robot.txt file, it is in the root dir.
I think this is stopping the site from being deep crawled.

I would appreciate any input
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post November 19th, 2004, 9:48 am

  • FatMouth
  • Graduate
  • Graduate
  • User avatar
  • Joined: Oct 31, 2004
  • Posts: 130
  • Status: Offline

Post November 19th, 2004, 12:13 pm

If it's for the site in your profile, there is no robots.txt file there.
  • KozSEO
  • Newbie
  • Newbie
  • No Avatar
  • Joined: Nov 19, 2004
  • Posts: 8
  • Loc: USA
  • Status: Offline

Post November 19th, 2004, 12:20 pm

no it is for this site

http://www.unclecharleyssausage.com/robots.txt

KOZ
  • FatMouth
  • Graduate
  • Graduate
  • User avatar
  • Joined: Oct 31, 2004
  • Posts: 130
  • Status: Offline

Post November 19th, 2004, 12:30 pm

Hmmm, I don't see why they would be getting a '404' then. Maybe someone else can shed some light on this.

BTW- cool looking site!
  • zcorpan
  • Student
  • Student
  • No Avatar
  • Joined: Nov 10, 2004
  • Posts: 77
  • Status: Offline

Post November 21st, 2004, 10:10 am

http://www.robotstxt.org/wc/norobots.html
Quote:
The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
You could delete your robots.txt if you want to allow them to index your site...

Page 1 of 1

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 19 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.