Back To Website Hosting Forum

Yahoo, Google, Msn Can't Crawl my Server!

  • twalters84
  • Graduate
  • Graduate
  • No Avatar
  • Joined: Sep 27, 2007
  • Posts: 161
  • Loc: Mount Savage, MD
  • Status: Offline

Post February 22nd, 2009, 10:01 pm

Greetings,

I have a severe problem with my server. I noticed that Yahoo, Google, and other search engines are not indexing my website.

http://www.green-watch.org

I wrote to Yahoo support about my problem and got the following message:

Quote:
Hello Travis, Thanks for writing to the Yahoo! Search and Directory Support. I understand we only have the index page for your site indexed. I have checked on your site and it appears that we're unable to crawl your site properly. We seem to get an assortment of timeouts and 500 server errors. You may want to check with your hosting company if they are blocking web crawlers or any other Yahoo! IPs or if they are aware of any other issues with your hosting. I apologize for the inconvenience. Thank you again for writing to the Yahoo! Search and Directory Support. Regards, Sky


Why would my server be behaving like this? I am on a dedicated server so nothing should be timing out.

Please assist me with fixing this. Thanks for any support and information that could be used to help my situation.

Sincerely,
Travis Walters
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post February 22nd, 2009, 10:01 pm

  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Joined: May 28, 2003
  • Posts: 23407
  • Loc: Woodbridge VA
  • Status: Offline

Post February 23rd, 2009, 5:48 am

Perhaps something in the DNS issues and Connection issues of this article might give you some ideas
http://www.coldfusionmuse.com/index.cfm ... leshooting
"There's no place like 127.0.0.1 except for ::1."
Alexandria Networks. Leader in IT consulting for associations/non-profits, and small to medium sized businesses around the northern Virginia and Washington D.C. metro area.
  • Kaumil
  • Born
  • Born
  • No Avatar
  • Joined: Feb 24, 2009
  • Posts: 4
  • Status: Offline

Post February 24th, 2009, 5:46 pm

I would double check all of the permissions on the files uploaded to the server. Then try to index the site again. Obviously, they will not index the pages if there are errors.
  • twalters84
  • Graduate
  • Graduate
  • No Avatar
  • Joined: Sep 27, 2007
  • Posts: 161
  • Loc: Mount Savage, MD
  • Status: Offline

Post February 24th, 2009, 9:29 pm

If there was a permission problem, would I see that when I tried to access it with a web browser?

I am not sure what to do.
  • twalters84
  • Graduate
  • Graduate
  • No Avatar
  • Joined: Sep 27, 2007
  • Posts: 161
  • Loc: Mount Savage, MD
  • Status: Offline

Post February 27th, 2009, 1:32 pm

Hey there,

Well this problem was a headache but I think I finally figured part of it out.

Code: [ Select ]
<cfset myURL = 'http://www.#myDomain#/scripts/getHostname.php?ipAddress=#cgi.REMOTE_ADDR#'>
    
<cfhttp method="Get" url="#myURL#" resolveurl="Yes" />
<cfset myHostName = '#Trim(cfhttp.FileContent)#'>
    
<cfset ipAddress = '#Replace(cgi.REMOTE_ADDR, ".", "-", "ALL")#'>
<cfset myHostName = '#Replace(myHostName, ipAddress, "*", "ALL")#'> 

<cfset setHostSession = 1>
  1. <cfset myURL = 'http://www.#myDomain#/scripts/getHostname.php?ipAddress=#cgi.REMOTE_ADDR#'>
  2.     
  3. <cfhttp method="Get" url="#myURL#" resolveurl="Yes" />
  4. <cfset myHostName = '#Trim(cfhttp.FileContent)#'>
  5.     
  6. <cfset ipAddress = '#Replace(cgi.REMOTE_ADDR, ".", "-", "ALL")#'>
  7. <cfset myHostName = '#Replace(myHostName, ipAddress, "*", "ALL")#'> 
  9. <cfset setHostSession = 1>


My name in the code above is green-watch.org.

It seems this code works for normal users but not for web crawlers.

Does anybody know why this could be?

Sincerely,
Travis Walters

Page 1 of 1

Post Information

  • Total Posts in this topic: 5 posts
  • Users browsing this forum: No registered users and 31 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.