Yahoo, Google, Msn Can't Crawl my Server!

  • twalters84
  • Graduate
  • Graduate
  • twalters84
  • Posts: 161
  • Loc: Mount Savage, MD

Post 3+ Months Ago

Greetings,

I have a severe problem with my server. I noticed that Yahoo, Google, and other search engines are not indexing my website.

http://www.green-watch.org

I wrote to Yahoo support about my problem and got the following message:

Quote:
Hello Travis, Thanks for writing to the Yahoo! Search and Directory Support. I understand we only have the index page for your site indexed. I have checked on your site and it appears that we're unable to crawl your site properly. We seem to get an assortment of timeouts and 500 server errors. You may want to check with your hosting company if they are blocking web crawlers or any other Yahoo! IPs or if they are aware of any other issues with your hosting. I apologize for the inconvenience. Thank you again for writing to the Yahoo! Search and Directory Support. Regards, Sky


Why would my server be behaving like this? I am on a dedicated server so nothing should be timing out.

Please assist me with fixing this. Thanks for any support and information that could be used to help my situation.

Sincerely,
Travis Walters
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Posts: 23456
  • Loc: Woodbridge VA

Post 3+ Months Ago

Perhaps something in the DNS issues and Connection issues of this article might give you some ideas
http://www.coldfusionmuse.com/index.cfm ... leshooting
  • Kaumil
  • Born
  • Born
  • Kaumil
  • Posts: 4

Post 3+ Months Ago

I would double check all of the permissions on the files uploaded to the server. Then try to index the site again. Obviously, they will not index the pages if there are errors.
  • twalters84
  • Graduate
  • Graduate
  • twalters84
  • Posts: 161
  • Loc: Mount Savage, MD

Post 3+ Months Ago

If there was a permission problem, would I see that when I tried to access it with a web browser?

I am not sure what to do.
  • twalters84
  • Graduate
  • Graduate
  • twalters84
  • Posts: 161
  • Loc: Mount Savage, MD

Post 3+ Months Ago

Hey there,

Well this problem was a headache but I think I finally figured part of it out.

Code: [ Select ]
<cfset myURL = 'http://www.#myDomain#/scripts/getHostname.php?ipAddress=#cgi.REMOTE_ADDR#'>
    
<cfhttp method="Get" url="#myURL#" resolveurl="Yes" />
<cfset myHostName = '#Trim(cfhttp.FileContent)#'>
    
<cfset ipAddress = '#Replace(cgi.REMOTE_ADDR, ".", "-", "ALL")#'>
<cfset myHostName = '#Replace(myHostName, ipAddress, "*", "ALL")#'> 

<cfset setHostSession = 1>
  1. <cfset myURL = 'http://www.#myDomain#/scripts/getHostname.php?ipAddress=#cgi.REMOTE_ADDR#'>
  2.     
  3. <cfhttp method="Get" url="#myURL#" resolveurl="Yes" />
  4. <cfset myHostName = '#Trim(cfhttp.FileContent)#'>
  5.     
  6. <cfset ipAddress = '#Replace(cgi.REMOTE_ADDR, ".", "-", "ALL")#'>
  7. <cfset myHostName = '#Replace(myHostName, ipAddress, "*", "ALL")#'> 
  8. <cfset setHostSession = 1>


My name in the code above is green-watch.org.

It seems this code works for normal users but not for web crawlers.

Does anybody know why this could be?

Sincerely,
Travis Walters

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 13 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.