Robots.txt

September 21st, 2005, 7:05 pm

does anyone have a tutorial on how to setup a robots.txt file onto your server...

September 21st, 2005, 8:07 pm

Here is a good one.

http://www.searchengineworld.com/robots ... torial.htm


If you can specify your exact needs, then i can provide content for your rebots.txt

Generally, you create file with .txt extension (user notepad or ms words to do that), you name file 'robots', then you write rules for search engine spiders and then you upload it to the root of your server.

[ATNO/TW]

September 21st, 2005, 8:29 pm

Yep. That's the one I learned from. Make sure you upload it in your root.

September 21st, 2005, 8:55 pm

Hello,

Thanks guys. Can you just give me some hint of "Rules".

thank you

Nicky

September 21st, 2005, 9:00 pm

Hello,
I have a page called error.aspx i do not want bot to visit it so can i write something like this?

User-agent: *
Disallow: error.aspx

At the same time i want it to visit my sitemap.xml and other pages of my site.

Is this ok guys?

Leme know

Chou

September 21st, 2005, 9:32 pm

Hi Shrek,

Check if this is correct

http://www.jnbsilver.com/robots.txt


chou

September 22nd, 2005, 12:08 am

It should be

User-agent: *
Disallow: /error.aspx/

Slash is very important!