Please help with Robots.txt

  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi Guys

Hope you can help.

I have just registered to use the google webmaster tools.

Under Robots.txt analysis (Analysis of cached robots.txt)

robots.txt URL http://www.alpine-connection.com/robots.txt
Last downloaded August 18, 2007 11:12:44 PM PDT
Status 404 (Not found) [?]

Is Status 404 a problem. I have nothing in my HTML code about robots should i place some code and if so what should it say

Regards

Paolo
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • coolslko
  • Proficient
  • Proficient
  • coolslko
  • Posts: 288
  • Loc: India

Post 3+ Months Ago

Hi,

Add below to your site

<meta name="robots" content="index,follow" />
  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi

Many thanks for the info

Regards

Paolo
  • batu544
  • Beginner
  • Beginner
  • batu544
  • Posts: 37
  • Loc: India

Post 3+ Months Ago

Hi,
If you want to exclude certain pages being indexed by google then you should put

<meta name="robots" content="noindex,nofollow" />




Its always better to put one robots.txt and controll the crawl of the search engines. :D


thanks,
batu544,
http://www.justclick2go.com
  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi There

What do you mean that it better to controll the search of the engines

Regards

Paolo
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

You should do a quick google search for robots.txt because it is a text file that you place in the main directory of your webserver, same place as your default.htm file

It is used for controling the spiders that crawl your site so that you can stop crawlers accessing information they shouldnt and also to help guide the spiders around your site, especially showing spiders not from yahoo or google where to find your sitemap.xml file.

put this in the meta tags
<meta name="robots" content="index,follow" />

then in your webserver directory make a file called robots.txt and put something like this inside it

User-agent: *
Sitemap: http://www.yourdomain.com/sitemap.xml

This tells spiders where to find your sitemap file, you can also allow / disallow certain directories like so

Disallow: /PDF/
But do a search and see how it works, you can specify different rules for different spider bots.

I cant find the link in may favourites, but a google search will show you, there is this site that will automatically check your robots.txt file and make sure that you dont have any errors in it. make sure you get the one that can recognise the sitemap.xml command.

also, that google error that you have is because you havnt made a robots.txt file its telling you that it cant find one, error 404 means doesnt exist.

hope this helps.
  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi Guys

Thanks for all your help

I placed the robots meta tag about 6 days ago,but i have just been to the google webmasters tools and it still showing 404 not found ( Last caculated 20 August)

Is this still a problem .

Can some one check to make sure i have placed the robots in the right place

http://www.alpine-connection.com

Regards

Paolo
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

Steven D wrote:
also, that google error that you have is because you havnt made a robots.txt file its telling you that it cant find one, error 404 means doesnt exist.

hope this helps.


The meta tag on each page, just tells spiders if they can crawl that page, and if they should follow links from that page or not.

People might disable it if they are a .edu or .govt site and dont want people getting free PR from their site.

The 404 Error that you are getting is because. you do not have a valid robots.txt file in your home directory of your website.

eg, if you type in

http://www.alpine-connection.com/robots.txt

you get

Not Found
The requested URL /robots.txt was not found on this server.

Apache/1.3.29 Server at alpine-connection.com Port 80

which is a 404 error. You have to make a robots.txt file.
  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi Steve

Thanks for your advice , is this straight forward to do to set up a Robots.txt.file

If you can point me to the right direction or point me to a website.

Having a 404 error does it stop you from getting rank higher in the search engines

Regards

Paolo
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

no i doubt it will stop you, your PR and SERP is determined by quality of information and incoming links and a few others, I think what happens is that robots check it to see if they are not allowed in certain directories, and if there isnt a robots.txt file they prob just go hard.


Just make a text file in your root directory, call it robots.txt and slap this in it

User-agent: *
Sitemap: http://www.yourdomainname.com/sitemap.xml
Allow: /


This will let every bot search every page and fdirectory of your site, this means that your images may appear in search engines, same with your flash objects and PDF files.

If there is a directory or directories you dont want to be searched just swap the allow line to
Disallow: /pics/

or

Disallow: /pics/
Disallow: /PDF/

What ever you need to block.

then go here
http://tool.motoricerca.info/robots-checker.phtml
and it will check if your file is up to scratch
  • bayliner75
  • Student
  • Student
  • bayliner75
  • Posts: 78

Post 3+ Months Ago

Hi Steve

Thanks for the info

Regards

Paolo
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

np budy, just like to try help and get things right, since I uploaded my site about 2 weeks ago I have spent over 1 full week changing it to be more SEO compatible, I have had to do the following,

Title Tag is not first 65 characters of the heading of an article
Meta Tags & Description Tags are now stored in the database for each article
Robots.txt file
Put all my JS code into seperate files and call when needed
Remove heaps of my tables, I had so many that it couldn't find parts of my site
Added a huge blurb at the bottom of important pages that explains what we do and the areas we help, so will get better SERP
Have listed in like 250 web directories, only 15 had accepted my link so far
Have uploaded 1 article to link 30 places, accepted in like 5 places so far
Registered for like 3 sets of forums and posted fanatically, but helpful *plum* so my posts dont get deleted.
Help some other guy set up some forums in return for featured link

and the list goes on and on, and I reakn I have like another 5 months of doing this before I get even close to a good ranking :(

Post Information

  • Total Posts in this topic: 12 posts
  • Users browsing this forum: No registered users and 10 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.