Robots Exclusion Protocol (REP)-List of Directives

  • coolslko
  • Proficient
  • Proficient
  • coolslko
  • Posts: 288
  • Loc: UK

Post 3+ Months Ago

According to recent post at google webmaster blog, Prashanth Koppula, Product Manager from Google has explained about improving robots exclusion protocol(REP)

Google, in the spirit of making the lives of webmasters simpler, releasing detailed documentation about how they implement REP. This will provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.

Common REP Directives
The following list are all the major REP features currently implemented by Google, Microsoft, and Yahoo!. With each feature, you'll see what it does and how you should communicate it.

Each of these directives can be specified to be applicable for all crawlers or for specific crawlers by targeting them to specific user-agents, which is how any crawler identifies itself. Apart from the identification by user-agent, each of our crawlers also supports Reverse DNS based authentication to allow you to verify the identity of the crawler.

1.Robots.txt Directives

(i)Disallow -Tells a crawler not to index your site -- your site's robots.txt file still needs to be crawled to find this directive, however disallowed pages will not be crawled.

'No Crawl' page from a site. This directive in the default syntax prevents specific path(s) of a site from being crawled.

(ii)Allow -Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow .
This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed except for a small section within it.

(iii)Sitemaps Location -Tells a crawler where it can find your Sitemaps.
Point to other locations where feeds exist to help crawlers find URLs on a site.

Please visit for complete information about robots exclusion protocol(REP) at-
http://googlewebmastercentral.blogspot. ... tocol.html

Post Information

  • Total Posts in this topic: 1 post
  • Users browsing this forum: No registered users and 1 guest
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum

© 1998-2017. Ozzu® is a registered trademark of Unmelted, LLC.