Your Robots.txt file is invalid

  • ccb056
  • Graduate
  • Graduate
  • User avatar
  • Joined: Mar 20, 2004
  • Posts: 189
  • Status: Offline

Post March 28th, 2004, 9:36 pm

I went to google and did a search for "site:www.ozzu.com" alot of the results you had tried to block using your robots.txt file, I then went to google's <a href="http://services.google.com:8882/urlconsole/controller?cmd=reload&lastcmd=login">Remove URL</a> section to see why the pages were indexed but not blocked because of the robots.txt, to make a long story short, google doen't seem to like the wildcards placed in the robots.txt.
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post March 28th, 2004, 9:36 pm

  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Joined: Dec 20, 2002
  • Posts: 8934
  • Loc: Seattle, WA & Phoenix, AZ
  • Status: Offline

Post March 29th, 2004, 12:59 pm

Hmm, I do not know any other way that I could block certain types of pages to be crawled that have the same prefix. Google doesn't support any kind of wildcards in the robots.txt file? That doesn't seem right to me.
Ozzu Hosting - Want your website on a fast server like Ozzu?
  • ccb056
  • Graduate
  • Graduate
  • User avatar
  • Joined: Mar 20, 2004
  • Posts: 189
  • Status: Offline

Post March 29th, 2004, 1:11 pm

User-agent: *
Disallow: /admin/
Disallow: /common
Disallow: /config
Disallow: /db/
Disallow: /faq
Disallow: /files/
Disallow: /groupcp
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /login
Disallow: /memberlist
Disallow: /mforum
Disallow: /modcp
Disallow: /ntopic
Disallow: /posting
Disallow: /privmsg
Disallow: /profile
Disallow: /ptopic
Disallow: /search
Disallow: /sutra
Disallow: /templates/
Disallow: /viewtopic
Disallow: /viewforum
Disallow: /viewonline

Post Information

  • Total Posts in this topic: 3 posts
  • Users browsing this forum: No registered users and 10 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.