Robots.txt

  • demonmaestro
  • Gold Member
  • Gold Member
  • User avatar
  • Posts: 674
  • Loc: Conroe, Texas

Post 3+ Months Ago

From what i can figure out its for the search engines but i am not sure what to put onto a robots.txt file. also how do i stop the search engine from going to directorys that i dont want people knowing?
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

Its pretty simple if you want to block all robots.

Just put a User-agent: * at the top of your robot.txt file, and then use the Allow/Disallow keywords.

Code: [ Select ]
User-agent: *
Disallow: /private
Disallow: /files
Allow: /files/upload.php
Disallow: search.php
  1. User-agent: *
  2. Disallow: /private
  3. Disallow: /files
  4. Allow: /files/upload.php
  5. Disallow: search.php


The above will tell robots to not go looking through the private folder or the files folder, but it will tell them to look at the upload.php file inside of the files folder. It also tells them they are not to look at the search.php page.
  • demonmaestro
  • Gold Member
  • Gold Member
  • User avatar
  • Posts: 674
  • Loc: Conroe, Texas

Post 3+ Months Ago

how ever if i do disallow: /private if someone wants to they can still view the robots.txt and see that i dont people going into that. right?
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

Ya, the robot text isn't to stop people from viewing your files, its just to stop robots, such as google, from indexing files you don't want to be indexed on google. Now if you really wanted to stop people from viewing your robot.txt file you could rewrite the url to a php file and check the user-agent there. Then if someone that's not a bot wouldn't be able to even see see the contents.

Generally you don't use a robot.txt file to protect files.

For example I have a folder I use to store general random junk. Basically my own file hosting site. I really don't care if someone wonders over and downloads something from it or views some images in it but I dont really want google to be bring up the images as they have nothing to do with my site. I also disallow robots from indexing my beta releases of sites. I want them to be viewable to the public but I dont want google to index them quite yet as they are not finished.

I also generally don't want google to index my search results page.
  • demonmaestro
  • Gold Member
  • Gold Member
  • User avatar
  • Posts: 674
  • Loc: Conroe, Texas

Post 3+ Months Ago

now can robots decript php scripts? like a database script?
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

DO you mean can they read the source of a php script? No, php scripts are interpreted on the server and then apache (or what ever your using) sends the output to the browser/robot.
  • demonmaestro
  • Gold Member
  • Gold Member
  • User avatar
  • Posts: 674
  • Loc: Conroe, Texas

Post 3+ Months Ago

Thanks Spoof. I am sure this will help out others also.!

Post Information

  • Total Posts in this topic: 7 posts
  • Users browsing this forum: No registered users and 7 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.