robots

June 25th, 2009, 10:00 am

I have just upload my site to the web, yay, im so proud, I know ive not done much but never mind!
Anyhow ive just thorght that there a few pages that realy dont need to be indexed now i know about the

Code: [ Download ] [ Select ]

<meta name="robots" content="noindex" />

but my site is run through php to get the pagers so and the robots code has to be in the head of the site so how do I get the code to show when I want it to and not when I don't Its probley very simple and im probley missing something fundamental here but just cope with me, thanks.

June 26th, 2009, 6:52 am

Make a file and name it as http://www.yoursite.com/robots.txt and upload it on the server. In the Robots.txt file write the pages which you don't want search engines to crawl. Use these text-

User-agent: *
Disallow: /pagename
Disallow: /pagename

June 29th, 2009, 12:24 am

Yep, what he said. Setting robot no follows within individual pages makes for nothing more than an un-manageable mess.

July 29th, 2009, 11:38 am

dhanjit wrote:

User-agent: *
Disallow: /pagename
Disallow: /pagename

yep... This is the best way to prevent specific pages to get crawled by search engiens.

[UPSGuy]

July 29th, 2009, 11:59 am

Are you two the yes-men or what? Solution! Concur! Yep!