Starting your own Search Engine

  • Marcus
  • Beginner
  • Beginner
  • User avatar
  • Posts: 35
  • Loc: NYC

Post 3+ Months Ago

I have always wanted to own my own search engine and mentioned it to a friend a while back. He recently told me he setup snuka.com and said let's see what you can do. He says here is your early xmas gift, so I what I was wondering if anyone had any thoughts on what it takes to make a good search engine.

1. What's the best way to get recurring traffic to my search engine? And to get other people to link to it?

2. What directories would one submit it to?

3. As it gets huge with content, how do i do quality control? Do i get volunteers? How do I know or protect the site from getting padded with editors business sites?

4. What makes the best engines? for instance, how should I weight link popularity? number of pages on a site, quality of that site? Should I rank the sites on quality of content, relevancy of content? How do the big engines do this with such massive amounts of websites?

5. What % of all searches on the net are done on google, msn, yahoo, aol etc..? Individually and as a whole?

6. WHich search engines provide the best ROI for customers? I find that google.com doesnt perform so well regarding adsense.
  • Thanol
  • Beginner
  • Beginner
  • Thanol
  • Posts: 38

Post 3+ Months Ago

Umm... it took years and large staffs to make Google, Inktomi, and FAST as good as they are today. If you plan on coding a good search engine you had better have A LOT of paitentence, money, and time on your hands.

Also #3 is out of place that question makes it sound like you're trying to make a web directory. There are no editors in search engines. A search engine is automated while a directory is human powered.
  • pompei
  • Graduate
  • Graduate
  • pompei
  • Posts: 117

Post 3+ Months Ago

I would say, if you want to do a search engine, then aim it at a niche market and get results coming up that are REALLY good for that particular niche, better than any other search engine out there. At this point, I definately wouldn't go for an all-encompassing SE and compete head-on with Google, Yahoo, and MSN...
  • john5269
  • Graduate
  • Graduate
  • john5269
  • Posts: 198

Post 3+ Months Ago

The best thing to do is try and make the search engine/directory as best as you can. As you start getting people to list their sites within your search engine/directory, then you should get a better idea of how you could improve it and make things better.

When making a site for the first time, it is hard to get it exactly how you would like it for the first time. You will always find ways of improving your site as time goes by.

As longs as your search engine/directory has good relevant and quick results then you should do alright. You will have to also advertise your search engine/directory well if you want to try and compete with Yahoo!, Google, etc.

If you want repeated visitors to your search engine/directory, then it would be best if you offered other things to your visitors like a games area, chatrooms, news, stocks and shares, email, etc. This will help you to get repeated visitors and while they are doing somthing on your site like playing games, or checking their email, then they may do a search while their there.
  • Bompa
  • Graduate
  • Graduate
  • User avatar
  • Posts: 229
  • Loc: Philippine Islands

Post 3+ Months Ago

I'll just say that I agree with pompei and john52, I think they both
gave common sense advice and there's no need for me to repeat it.

Bompa
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

Yeah, I've also been toying with the idea of setting up a search/directory site. As has been suggested earlier in this thread, the one I've been thinking about is aimed at a fairly specific but large audience, with a very select theme of website.

At least initially, it's going to need to be humanly administrated, I just don't have the knowledge yet of a search engine's inner-workings to be able to write the scripts for something fully automated. Also, given how many web pages there are on the net compared to the small percentage of them that I would like to index, it's going to waste MASSES of bandwidth just looking to see if they're on topic...

So I think human-admin'd is going to be the way to go for my idea for a long while yet.
  • john5269
  • Graduate
  • Graduate
  • john5269
  • Posts: 198

Post 3+ Months Ago

Axe,

A human edited search engine could take up alot of your time. As it is targeted to a specific audience, it may be best to have it human edited as you say. But if I was you, I would only use people to find and add the listings.

I would then use some sort of software/scripts that will detect when the sites url's within your search engine becomes broken, returns 404 errors, etc.

Using this software will improve your site as people won't be clicking on dead links all the time and it will be easier and quicker to maintain your site.
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

Yup, even though human submitted & verified (at least that sites exist and are on topic), there will be SOME automated scripts to sniff the rest of the pages on those sites, and check for dead links, etc., but it will not go off and seek new sites on its own or go above the directory name or the URL that was submitted - for example, if somebody has a related site hosted on say geocities or tripod that they submitted for addition, if it were to be approved, the scripts won't just go and look for any page it can find on that domain, it will only look for files & directories that exist within the directory that was submitted. If somebody were to submit a complete domain name, with no subdirectory, the script would seek to index all pages it can find on that domain name.

This project, even with a limited audience is a HUGE undertaking, and my plans are still 99% theory. I already have a nice sized database of sites, that will total about 10,000 pages on many subjects in the field to get things started, but this is nothing compared to what exists on this topic - but then, given how long the Internet's been around, and the amount of people who have pages up these days about almost anything, I don't suppose there's a topic out there that doesn't have hundreds of thousands of pages related to it.

It's going to require a lot of bandwidth & servers to run at any sort of a half-decent speed, which is going to mean investors to get things rolling, and advertisers to pay the investors back once things are rolling. This is, of course, assuming things will get rolling. They may not, so this is something I still have to look into.

Even if it all eventually comes together, it's going to take a while, and something else may turn up in the meantime that offers more return for less effort. But, even if it turns out not to be a viable prospect, I'm still going to work on some scripts, and get something basic completed for myself, if only to research more how on search engines actually work.
  • Johan007
  • Guru
  • Guru
  • User avatar
  • Posts: 1080
  • Loc: Aldershot, UK

Post 3+ Months Ago

I have had a small experience in putting together an search engine aimed at school teachers where they can search for teaching resources and materials as well as research..

We had a small team 1 coder and a few other people assigned to this project and in the end it was scrapped cos not many people used it.

We found that we got best results spidering only a few websites. These would include sites with a lot of content including all the newspaper and news sites like the BBC, CNN. Also scientific journals.

This way we did not have to worry too much about broken links due to lack of staff.

It was not easy as our programmer was good he could not devote enough time. Often a search would bring up many pages from one website but if given enough time I think the product would have been good. - just don’t think its very profitable.

Post Information

  • Total Posts in this topic: 9 posts
  • Users browsing this forum: No registered users and 3 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.