Google Indexing Nonexistent Files

  • Tandem
  • Born
  • Born
  • No Avatar
  • Joined: Apr 11, 2009
  • Posts: 3
  • Status: Offline

Post April 11th, 2009, 1:44 pm

My Error Logs show quite a few 404 entries.
The referer is usually google.gr, google.ee google.com.br etc.

Please keep in mind, this is not a case of broken links, renamed or removed files. The files in question never existed on the sites (as far as I know).

Also, the sites and the directories that the indexes point to have all robots.txt files with the following:
User-agent: *
Disallow: /

The sites are for private use and are not indexed by SEs. I am aware that bots can ignore the robots.txt files.

What concerns me is that the file names usually are something along the lines:
....serial-free.html
....CD-key-changer.html
...something-sex.html and so on.

Does anyone have any ideas about what's is going on? How do these end up in google index?

Thank you for your time.
  • Anonymous
  • Bot
  • No Avatar
  • Joined: 25 Feb 2008
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post April 11th, 2009, 1:44 pm

  • Don2007
  • Web Master
  • Web Master
  • No Avatar
  • Joined: Nov 21, 2006
  • Posts: 4924
  • Loc: NY
  • Status: Offline

Post April 11th, 2009, 7:52 pm

It sounds like a bot looking for cracks. Don't worry about it.
How do you know when a politician is lying? His mouth is moving.
  • Tandem
  • Born
  • Born
  • No Avatar
  • Joined: Apr 11, 2009
  • Posts: 3
  • Status: Offline

Post April 11th, 2009, 8:54 pm

That's what it looks like, but the referer is google.
  • Don2007
  • Web Master
  • Web Master
  • No Avatar
  • Joined: Nov 21, 2006
  • Posts: 4924
  • Loc: NY
  • Status: Offline

Post April 11th, 2009, 9:23 pm

Let's say I type into the google search box

site:tandemsite.com inurl:/CD-key-changer.html & press enter, wouldn't that make the google the referrer? That's what I think is happening but not by hand.

See if you can find googlehacking.pdf

johnny I hack stuff seems to be down for the moment but the pdf should still be available somewhere.
How do you know when a politician is lying? His mouth is moving.
  • joebert
  • Sledgehammer
  • Genius
  • No Avatar
  • Joined: Feb 10, 2004
  • Posts: 13455
  • Loc: Florida
  • Status: Offline

Post April 12th, 2009, 3:25 am

Tandem wrote:
That's what it looks like, but the referer is google.


The referer by itself isn't of much importance. Check the IP address against a list of known Google networks to confirm.

I can send a request to the site looking for "/i-was-here.joebert" that appears to come from Google right now if you send me the address to one of the sites in question, assuming you only consider the referer.
Referers can be forged, IP addresses can kinda be spoofed via proxy, but there's an extremely tiny, right next to non-existant, chance that an IP address pointing back to Googles network can be forged.

Are you sure they're "in Googles index", did you look for them by searching Google ?
Or did you just assume they're in the index because of the referer in your own logs ?
Strong with this one, the sudo is.
  • Tandem
  • Born
  • Born
  • No Avatar
  • Joined: Apr 11, 2009
  • Posts: 3
  • Status: Offline

Post April 12th, 2009, 11:01 am

Thank you all for your input.

Yes, the referer was google, I verified the index.

This is the latest entry:
Code: [ Select ]
[Sun Apr 12 06:26:29 2009] [error] [client 78.160.xxx.xxx] File does not exist: /home/SITENAME/public_html/DIR/Smileys, referer: http://www.google.com.tr/search?hl=tr&r ... edir&meta=


My site is no longer among the results because I removed it from the index using Google's Remove URLs tool.
  • joebert
  • Sledgehammer
  • Genius
  • No Avatar
  • Joined: Feb 10, 2004
  • Posts: 13455
  • Loc: Florida
  • Status: Offline

Post April 12th, 2009, 2:18 pm

That really is strange.
Strong with this one, the sudo is.

Post Information

  • Total Posts in this topic: 7 posts
  • Users browsing this forum: No registered users and 43 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 2011 Unmelted, LLC. Ozzu® is a registered trademark of Unmelted, LLC.