Your Robots.txt file is invalid

  • ccb056
  • Graduate
  • Graduate
  • User avatar
  • Posts: 189

Post 3+ Months Ago

I went to google and did a search for "site:www.ozzu.com" alot of the results you had tried to block using your robots.txt file, I then went to google's <a href="http://services.google.com:8882/urlconsole/controller?cmd=reload&lastcmd=login">Remove URL</a> section to see why the pages were indexed but not blocked because of the robots.txt, to make a long story short, google doen't seem to like the wildcards placed in the robots.txt.
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Posts: 9090
  • Loc: Seattle, WA & Phoenix, AZ

Post 3+ Months Ago

Hmm, I do not know any other way that I could block certain types of pages to be crawled that have the same prefix. Google doesn't support any kind of wildcards in the robots.txt file? That doesn't seem right to me.
  • ccb056
  • Graduate
  • Graduate
  • User avatar
  • Posts: 189

Post 3+ Months Ago

User-agent: *
Disallow: /admin/
Disallow: /common
Disallow: /config
Disallow: /db/
Disallow: /faq
Disallow: /files/
Disallow: /groupcp
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /login
Disallow: /memberlist
Disallow: /mforum
Disallow: /modcp
Disallow: /ntopic
Disallow: /posting
Disallow: /privmsg
Disallow: /profile
Disallow: /ptopic
Disallow: /search
Disallow: /sutra
Disallow: /templates/
Disallow: /viewtopic
Disallow: /viewforum
Disallow: /viewonline

Post Information

  • Total Posts in this topic: 3 posts
  • Users browsing this forum: No registered users and 3 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.