robots.txt

  • KozSEO
  • Newbie
  • Newbie
  • KozSEO
  • Posts: 8
  • Loc: USA

Post 3+ Months Ago

My log files keep showing 404 errors on my robots.txt file
I had the file validated and it seems fine, here is the file

# Allows all robots

User-agent: *
Disallow:

I have just started using the robot.txt file, it is in the root dir.
I think this is stopping the site from being deep crawled.

I would appreciate any input
  • FatMouth
  • Graduate
  • Graduate
  • User avatar
  • Posts: 130

Post 3+ Months Ago

If it's for the site in your profile, there is no robots.txt file there.
  • KozSEO
  • Newbie
  • Newbie
  • KozSEO
  • Posts: 8
  • Loc: USA

Post 3+ Months Ago

no it is for this site

http://www.unclecharleyssausage.com/robots.txt

KOZ
  • FatMouth
  • Graduate
  • Graduate
  • User avatar
  • Posts: 130

Post 3+ Months Ago

Hmmm, I don't see why they would be getting a '404' then. Maybe someone else can shed some light on this.

BTW- cool looking site!
  • zcorpan
  • Student
  • Student
  • zcorpan
  • Posts: 77

Post 3+ Months Ago

http://www.robotstxt.org/wc/norobots.html
Quote:
The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
You could delete your robots.txt if you want to allow them to index your site...

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 3 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.