Robot.txt file

  • weblaunchphxx
  • Graduate
  • Graduate
  • User avatar
  • Posts: 143

Post 3+ Months Ago

The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below.

HTML meta tags for robots.
Code: [ Select ]
< meta name="robots" content="noindex,nofollow" />


To allows all robots
Code: [ Select ]
User-agent: *
Disallow:

To all robots out
User-agent: *
Disallow: /

To prevent pages from all crawlers
User-agent: *
Disallow: /page name/
  1. User-agent: *
  2. Disallow:
  3. To all robots out
  4. User-agent: *
  5. Disallow: /
  6. To prevent pages from all crawlers
  7. User-agent: *
  8. Disallow: /page name/



To prevent pages from specific crawler
Code: [ Select ]
User-agent: GoogleBot
Disallow: /page name/
  1. User-agent: GoogleBot
  2. Disallow: /page name/
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • batu544
  • Beginner
  • Beginner
  • batu544
  • Posts: 37
  • Loc: India

Post 3+ Months Ago

Oh ! thanks
  • tomfrederick
  • Novice
  • Novice
  • tomfrederick
  • Posts: 22

Post 3+ Months Ago

Thanks for nice information that is helpful to me.
  • terryson01
  • Novice
  • Novice
  • terryson01
  • Posts: 28

Post 3+ Months Ago

weblaunchphxx you give very nice information about robot.txt.

Anyone wants more information about robot.txt can read on http://www.seoconsultants.com/robots-text-file/

Post Information

  • Total Posts in this topic: 4 posts
  • Users browsing this forum: No registered users and 36 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.