getting CGI pages to be crawled

  • natural_angel33
  • Graduate
  • Graduate
  • User avatar
  • Joined: Feb 03, 2004
  • Posts: 127
  • Loc: Las Vegas, NV (blech!)
  • Status: Offline

Post February 4th, 2004, 12:49 pm

I know that web spiders generally don't like CGI pages because of all the foreign characters in the URL address, so what would be a good way to make CGI pages more spider friendly?
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post February 4th, 2004, 12:49 pm

  • Axe
  • Genius
  • Genius
  • User avatar
  • Joined: Jan 07, 2004
  • Posts: 5744
  • Loc: Sub-level 28
  • Status: Offline

Post February 4th, 2004, 12:54 pm

Using mod_rewrite would probably be your best bet, of course, this would requiring modification of the script to output the "fake" URLs.

If you hit the search up top on this page, and enter mod_rewrite, there should be some useful posts. There's been a little discussion about this recently. Mostly with PHP based scripts, but I don't see why it wouldn't work with CGI script URLs too :)
  • natural_angel33
  • Graduate
  • Graduate
  • User avatar
  • Joined: Feb 03, 2004
  • Posts: 127
  • Loc: Las Vegas, NV (blech!)
  • Status: Offline

Post February 4th, 2004, 12:59 pm

Cool, thanks! :D
  • SEO Guy
  • Beginner
  • Beginner
  • User avatar
  • Joined: Feb 02, 2004
  • Posts: 47
  • Loc: Canada
  • Status: Offline

Post February 4th, 2004, 2:50 pm

For more info on mod_rewrite visit:
http://httpd.apache.org/docs/mod/mod_rewrite.html

We are developing a tutorial site based on mod rewrite as well but wont be launched for a good month but if you have any specific questions feel free to email me or seobob@seo-guy.com he is my lead programmer.

OF course there are always the helpful members here as well.

Mod rewrite kicks ASS!
  • tiekie
  • Novice
  • Novice
  • No Avatar
  • Joined: Mar 03, 2007
  • Posts: 30
  • Status: Offline

Post March 3rd, 2007, 8:01 am

Cheers Mate

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 17 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.