Google Bot Revisits

  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

I just want to outline my question and if I can get some feedback it would be appreciated.

I put up my site only a week ago, first thing I did was correct meta tags, almost all 100%, keywords, description and title.

I created my sitemap.xml and robots.txt and submittied my site to google.

All is fine. Then luckily less than 24 hours later google crawled my site for the first time. Again all is well.

On my site, each time a page loads, I record the user IP, time, date, page name, referrer addr etc (I am very anal about marketing and scorekeeping)

1 day after google crawled my site, the whole site, when I went into webmaster tools it said you have been indexed on the 20th August 07, all is fine.

Since I was first crawled, I have modified my homepage to provide previews of my sites information (random selection) to give always fresh content on my main page. Then I allowed previews of the information I am selling, so that I can get part of my info added to google index, then I added a published articles section for the articles I have started publishing on the net. all is well.

Now since google crawled my site, and I made those changes, googlebot has crawled my site 4 more times but my changes are not reflected in the search results, and the cached pages are still my old pages.

What I want to know is, how long after a crawl is the information updated in the index (First time I was crawled took another 24 hours before I was indexed) as it has been a few days now.

Also, when i run the link:yourdomain.com none of my external links are comming up, but yahoo, who I didnt submit a sitemap to until about 3 days ago has already indexed like 20 of the one way links to my site?

Do I have to wait until I am out of the google sandbox before I can use the link:domain.com search criteria.

thanks, sorry it is so long, just wanted to get it right.
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • coolslko
  • Proficient
  • Proficient
  • coolslko
  • Posts: 288
  • Loc: India

Post 3+ Months Ago

Well,
googlebot visits time to time to webpages, it doesn't mean googlebot index pages everytime. Often it happens that after visiting googlebot may be not index all pages because of any reason like crawling error or duplicate content or anything else but you will find after few visits your pages will be indexed.

As you said your site first time indexed within 24 hrs. it is just because it was fresh and google bot prefers to index fresh and quality webpages.

And as you said your links not appears on google but appears on yahoo...
I want to tell you that google shows the shortest no of links having enough good quality and google updates for backlinks takes place minimum 3-4 months and it happens before or around PR updation.

And yahoo is the search engine shows largest no backlinks...

So do not worry to much about indexing and links on google ...keep on updating your website and work on building quality links...
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

Thank you coolslko.

Just to confirm, the PR updates happens for the first time 3-4 months after first being indexed?

or

it happens every 3-4 on their own cycle not based on when the site went live?

thanks again.
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

K i am about to bust a nut. I have obviously messed something up because the bot just randomly does what it pleases.

I Will try an outline it a little more.

I made my sitemap, and inside my sitemap I used the priority tags to tell googlebot what is important what what is not.

Things like terms and conditions I left at 0.5, and things like my article pages I bumped up to 0.9
And set the changefrequency to daily

Just before I wrote the above post i was trying to get it to stop crawling my static pages and go onto my dynamic pages and so to make do what it is suppose to, I removed the meta tags from things like privacy policy etc, and removed them completely from my sitemap.

In the last 36 hours, googlebot has come back and crawled my terms and conditions, my privacy policy and my advertisers login page like 4 times each and it has only once ever in the history of my site (18 days) crawled my dynamic pages.

Is there something that I am missing, some secret little clue to make it stop crawling my static pages, there are no meta tags, its not in the sitemap, am I suppose to list it in my robots.txt file as a disalow or something, I want those pages to show up in search results, I have some links going to them, I just dont want google crawling them over my important pages.

thanks.
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

and this is the results from a google search of
site:http://www.walkoffwithyourcompetitorsmoney.com

Walk Off With Your Competitors Money - 3 visits - Aug 23
I removed the words
http://www.walkoffwithyourcompetitorsmoney.com/ - 30k - Cached - Similar pages -

Walk Off With Your Competitors Money
I removed the words
http://www.walkoffwithyourcompetitorsmo ... _terms.asp

Its putting my legal_terms and advertisers_default.asp & Legal_Legal.asp and legal_privacy.asp into the results and not displaying the pages that it should.

Seems bit unfair wouldnt you say, those pages dont even have meta tags and arnt in the sitemap.xml file
:(
  • Fergie
  • Student
  • Student
  • User avatar
  • Posts: 85
  • Loc: Canada

Post 3+ Months Ago

Google index everything. You should be okay.
  • Steven D
  • Proficient
  • Proficient
  • Steven D
  • Posts: 263

Post 3+ Months Ago

holy smokes what a hotty, looks like aviril levine.

Can you try and explain this to me then, I feel like a tool not being able to figure it out.

http://www.gritechnologies.com/tools/di ... browse=cat

This is the poodle technologies search crawler you use for examining my site, move your mouse over the link for click here to preview and the link it fuked, dont click it cuz it all turns into a mess.

Then go and visit that page on my site, its perfect, as if god created it himself. I have checked the code, its all fine
I removed the reference to sessions incase googlebot doesnt allow them, but still no dice.

Is there somewhere else I can try to view my site?

Post Information

  • Total Posts in this topic: 7 posts
  • Users browsing this forum: No registered users and 12 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.