robots.txt

  • KozSEO
  • Newbie
  • Newbie
  • KozSEO
  • Posts: 8
  • Loc: USA

Post 3+ Months Ago

My log files keep showing 404 errors on my robots.txt file
I had the file validated and it seems fine, here is the file

# Allows all robots

User-agent: *
Disallow:

I have just started using the robot.txt file, it is in the root dir.
I think this is stopping the site from being deep crawled.

I would appreciate any input
  • smo
  • Beginner
  • Beginner
  • smo
  • Posts: 43
  • Loc: India

Post 3+ Months Ago

KozSEO wrote:
I have just started using the robot.txt file, it is in the root dir.


Is it robot.txt or robots.txt ? It should be robots.txt

Your content is OK.
  • longcall911
  • Novice
  • Novice
  • longcall911
  • Posts: 23
  • Loc: Northern NJ

Post 3+ Months Ago

You can try browsing to it to be sure it exists.

http://www.yoursite.com/robots.txt

If it does, I would check with your hosting service to find out why it might be reporting 404s.
  • antiladybear
  • Novice
  • Novice
  • antiladybear
  • Posts: 20

Post 3+ Months Ago

Try rebuilding your robots.txt file at this address, it's a free to to use for robots. http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm

Post Information

  • Total Posts in this topic: 4 posts
  • Users browsing this forum: No registered users and 5 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.