Many more Googlebot hits than Yahoo ones?

  • lioness
  • Mastermind
  • Mastermind
  • User avatar
  • Posts: 1615

Post 3+ Months Ago

So far this month I have:
Code: [ Select ]
Googlebot    860 hits (+50 on robots.txt)
Inktomi Slurp   11 hits (+15 on robots.txt)
  1. Googlebot    860 hits (+50 on robots.txt)
  2. Inktomi Slurp   11 hits (+15 on robots.txt)

Can anyone explain (or provide a link to) why Yahoo's Inktomi Slurp can hit so many fewer pages than Googlebot? My robots.txt is simply:

Code: [ Select ]
User-agent: *
Disallow: /images/
  1. User-agent: *
  2. Disallow: /images/
  • DIYweb
  • Novice
  • Novice
  • DIYweb
  • Posts: 23

Post 3+ Months Ago

People still use Inktomi? I wouldn't really worry about it, who cares? What you have is probably fine, I mean how much spidering do they do now...

While Yahoo does own Inktomi, what I'd be much more concerned about is if the real / main Yahoo bot wasn't visiting your site much.

Do you also so an entry for
Yahoo! Slurp
  • lioness
  • Mastermind
  • Mastermind
  • User avatar
  • Posts: 1615

Post 3+ Months Ago

Hi DIYweb - thanks for your response. No I do not have any hits from 'Yahoo! Slurp' - only Google, MSN, Alexa, AskJeeves and several unidentified ones.

I do have pages indexed by Yahoo though - how is that?
  • DIYweb
  • Novice
  • Novice
  • DIYweb
  • Posts: 23

Post 3+ Months Ago

Uggh, I just closed the tab by mistake...

Ah, I'm not sure what to say. If you are talking about the cycle site, then you are excluding more directories, but I don't think that really matters.

I see a lot more pages indexed in Google than Yahoo, but as always, Yahoo is showing a lot more backlinks than Google.

Has the site been online since the domain was bought in the fall of 2004? Clearly it's been online for awhile at least, so these results are odd.
  • lioness
  • Mastermind
  • Mastermind
  • User avatar
  • Posts: 1615

Post 3+ Months Ago

DIYweb wrote:
Uggh, I just closed the tab by mistake...

Ah, I'm not sure what to say. If you are talking about the cycle site, then you are excluding more directories, but I don't think that really matters.

I see a lot more pages indexed in Google than Yahoo, but as always, Yahoo is showing a lot more backlinks than Google.

Has the site been online since the domain was bought in the fall of 2004? Clearly it's been online for awhile at least, so these results are odd.


Yes it is the cycle site.

The site has been online for approx a year, but I made major changes which have been in effect since Jan 1st of this year.

Not sure what to say myself. If I find anything out about how to get 'Yahoo! Slurp'-ed then I'll post it here.
  • DIYweb
  • Novice
  • Novice
  • DIYweb
  • Posts: 23

Post 3+ Months Ago

I doubt that the normal Yahoo listings are from Inktomi and I'm sure it's safe to say that Yahoo! Slurp has been there in January at least. If you made changes to the robots.txt then, maybe something weird has happened, but I'd say just give it some time and things will correct themselves.

As long as there have been some changes to the spiderable pages, the bots will come around and all will be well. It's good to make changes every now and then, even if there is nothing to add really... but hopefully you're adding fresh content anyway.

Post Information

  • Total Posts in this topic: 6 posts
  • Users browsing this forum: No registered users and 2 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
cron
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.