Back To Other Search Engines & Directories

Robot.txt file

  • weblaunchphxx
  • Graduate
  • Graduate
  • User avatar
  • Joined: Jul 31, 2007
  • Posts: 145
  • Status: Offline

Post December 12th, 2007, 11:48 pm

The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below.

HTML meta tags for robots.
Code: [ Select ]
< meta name="robots" content="noindex,nofollow" />


To allows all robots
Code: [ Select ]
User-agent: *
Disallow:

To all robots out
User-agent: *
Disallow: /

To prevent pages from all crawlers
User-agent: *
Disallow: /page name/
  1. User-agent: *
  2. Disallow:
  4. To all robots out
  5. User-agent: *
  6. Disallow: /
  8. To prevent pages from all crawlers
  9. User-agent: *
  10. Disallow: /page name/



To prevent pages from specific crawler
Code: [ Select ]
User-agent: GoogleBot
Disallow: /page name/
  1. User-agent: GoogleBot
  2. Disallow: /page name/
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post December 12th, 2007, 11:48 pm

  • batu544
  • Beginner
  • Beginner
  • No Avatar
  • Joined: Aug 20, 2007
  • Posts: 37
  • Loc: India
  • Status: Offline

Post December 19th, 2007, 12:47 am

Oh ! thanks
  • tomfrederick
  • Novice
  • Novice
  • No Avatar
  • Joined: Nov 30, 2007
  • Posts: 22
  • Status: Offline

Post February 4th, 2008, 10:32 pm

Thanks for nice information that is helpful to me.
  • terryson01
  • Novice
  • Novice
  • No Avatar
  • Joined: Jan 27, 2008
  • Posts: 28
  • Status: Offline

Post February 5th, 2008, 10:41 pm

weblaunchphxx you give very nice information about robot.txt.

Anyone wants more information about robot.txt can read on http://www.seoconsultants.com/robots-text-file/

Page 1 of 1

Post Information

  • Total Posts in this topic: 4 posts
  • Users browsing this forum: No registered users and 24 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.