(Q) Best ways to get a website spidered

  • Marqi
  • Newbie
  • Newbie
  • User avatar
  • Posts: 13
  • Loc: Detroit

Post 3+ Months Ago

Does anyone have any advice about the best way to get a website spidered by various search engines? I am a newbie to SEO/SEM but I am learning fast. I also am new to this message board, so any cool networking or help is very appreciated.

also., does the robots.txt have anything to do with it? I placed one on my server, but just used the "*" so all can spider.

Thank you in advance

Shannon
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

the best way to get spiders visiting your web site is to get lots of links pointing at it.

Spiders work by following links - so logic says the more links you have pointing at your site the better chance you have of being found by the search engines.
  • Marqi
  • Newbie
  • Newbie
  • User avatar
  • Posts: 13
  • Loc: Detroit

Post 3+ Months Ago

Thanx Jess, so does blogging and posting on forums count as well???
  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

Marqi wrote:
Thanx Jess, so does blogging and posting on forums count as well???


Absolutely :)

Make use of directories as well - there are some great directories that offer free links or one time payments.

http://www.blueangels.net is a good source for directories.
  • madmonk
  • Mastermind
  • Mastermind
  • madmonk
  • Posts: 2115
  • Loc: australia

Post 3+ Months Ago

A high activity site will get your site spidered fast too! :-)
  • rajputkashyap
  • Graduate
  • Graduate
  • User avatar
  • Posts: 198
  • Loc: USA

Post 3+ Months Ago

Hi,
one of my page is not being crawled since 9th march 2004, I have checked for errors with Webbug, but it shows OK 200., Well links from homepage and other pages..., other site pages crawl almost regularly

the page is : http://www.tvaproductions.com/html/body ... overs.html

Quote:
Jess, This is not the site I was talking about you


I have checked robots.txt, but there is no exclusion for this page

Cant get why this page is not being indexed

Please help

kashyap
  • GSlinger
  • Proficient
  • Proficient
  • User avatar
  • Posts: 384
  • Loc: Ohio

Post 3+ Months Ago

you can request it from google too. u fill out a form with your URL and they say googlebot "might" crawl it.

not all spiders reference their crawls using how many links point at your site. many do, but dont rely on links for that purpose. just look at the big search engine sites, most have a place where u can send them your url.
but if you have a site that is currently ranked relatively high, then a link to AND from it will affect your new sites ranking. i know i keep contradicting myself, but every search engine ranks differently and looks at different parameters. keep that in mind.
  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

Quote:
just look at the big search engine sites, most have a place where u can send them your url.


This is true - however its usually pointless submitting through them.

You can get a web site listed in google in 2-3 days by gaining links to the site. Try submitting to the add url form in google and don't set up any links and you could well be waiting months.
  • GSlinger
  • Proficient
  • Proficient
  • User avatar
  • Posts: 384
  • Loc: Ohio

Post 3+ Months Ago

back when i had my site it took em 3 hours.

im telling ya, links might bring your rank up with google, maybe google will even crawl it using the links, but google isnt the only search engine out there and with every update it gets worse and more unpredictable.
submit your site's URL. it took google 3 hours and then with no links goin to or from my site almost every other search enigine ranked me within a week.
  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

GSlinger wrote:
back when i had my site it took em 3 hours.

im telling ya, links might bring your rank up with google, maybe google will even crawl it using the links, but google isnt the only search engine out there and with every update it gets worse and more unpredictable.
submit your site's URL. it took google 3 hours and then with no links goin to or from my site almost every other search enigine ranked me within a week.


Any search engine worth being in has a spider that follows links to crawl your site. That is how spiders work - by following links from one site to another.

Whether you submit through the add url form or not - your going to need links to get any where with all the major engines.

My personal opinion is - I'd rather the engines find my site without me having to submit to them - because then they chose to list my pages - I didn't ask them too, and maybe it's just me - but I think that gives any new site an advantage.
  • GSlinger
  • Proficient
  • Proficient
  • User avatar
  • Posts: 384
  • Loc: Ohio

Post 3+ Months Ago

ok
i didnt mean to come off rude. everyone has their own way of doin things and neither is better or worse. im sure u have been doin this longer than me.
i didnt want this to become an argument Jess. my bad.

Marqi, these are just suggestions. u can do both if u want.
  • Jess
  • Guru
  • Guru
  • User avatar
  • Posts: 1153
  • Loc: USA

Post 3+ Months Ago

GSlinger wrote:
ok
i didnt mean to come off rude. everyone has their own way of doin things and neither is better or worse. im sure u have been doin this longer than me.
i didnt want this to become an argument Jess. my bad.

Marqi, these are just suggestions. u can do both if u want.


We are not arguing :)
Just discussing - thats what these boards are for.

I didn;t intend for that to sound heated - appologies :)
  • GSlinger
  • Proficient
  • Proficient
  • User avatar
  • Posts: 384
  • Loc: Ohio

Post 3+ Months Ago

Quote:
When a URL is submitted to Google, we look for it in our next crawl. If you've already submitted your URL, your site could easily appear in our new index, which will go up when the current crawl is completed.


Quote:
Google's robots jump from page to page on the Web via hyperlinks, so the more sites that link to you, the more likely it is that we'll find you quickly.


straight from Google. we are both right Jess. so link to other sites and submit it is what i would do. but remember Google is not the only search enigine out there. read my thread on Clusty. its a new search engine still in the beta. many ppl like it and some even think it will contend with Google.
  • ZenSEO
  • Student
  • Student
  • ZenSEO
  • Posts: 71

Post 3+ Months Ago

I agree that even just one link from a frequently visited/spidered site will do it. I have a pr5 informational site that's frequently updated and spidered and two other sites that are linked from it and they get spider hits often multiple times per day coming from my frequently spidered site through the links.... so not only is it the best way to get listed, it's the best way to keep the spiders to keep your content updated -- ie if i just submit it but i have no links, even if it got listed the spiders would have no reason to visit frequently in case i made updates to my SEO techniques or site updates... with a link from a frequently spidered site any updates I make to my site are picked up FAST... the true benefit :) not just being listed, but being spidered on a daily basis so that your listing is always up to date. My cache on google is almost always from the day before :)
  • fourthring
  • Graduate
  • Graduate
  • User avatar
  • Posts: 152
  • Loc: San Diego / Beijing

Post 3+ Months Ago

rajputkashyap wrote:
Hi,
one of my page is not being crawled since 9th march 2004, I have checked for errors with Webbug, but it shows OK 200., Well links from homepage and other pages..., other site pages crawl almost regularly

the page is : http://www.tvaproductions.com/html/body ... overs.html

Quote:
Jess, This is not the site I was talking about you


I have checked robots.txt, but there is no exclusion for this page

Cant get why this page is not being indexed

Please help

kashyap


Maybe it's a duplicate content issue? Session IDs? The log file is a great place to look. How deep is the page? Place a link from your highest page ranked page directly to that page and see what happens. Is it all the spiders or just certain ones? Usually you can see activity with at least one or two different bots - especially those pesky spam bots :x

Post Information

  • Total Posts in this topic: 15 posts
  • Users browsing this forum: No registered users and 3 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.