Sitemaps

  • rtchar
  • Expert
  • Expert
  • User avatar
  • Posts: 606
  • Loc: Canada

Post 3+ Months Ago

Google is recommending you start simple ... A text file or simple XML file with just a few pages listed are OK.

Don't worry if every page on your site is listed in your first attempts, this program will NOT replace the crawlers. Besides you can submit a new file with additions every day if you like.

The program is still in Beta, the point is to test the system before it goes live in unattended mode.

mighty b
Quote:
It was downloaded fine and the status is now OK.


According to Google OK means that the uploaded file processed correctly and is now queued. Of course there is no guarantee the "suggestions" will be included in the index. :lol:
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

  • allgoodpeople
  • Proficient
  • Proficient
  • User avatar
  • Posts: 379
  • Loc: here

Post 3+ Months Ago

just a thought . . .

if this takes off and gets to be standard practice with websites, will other search engines be able (or even be allowed) to access this config.xml file for their own indices?
  • rtchar
  • Expert
  • Expert
  • User avatar
  • Posts: 606
  • Loc: Canada

Post 3+ Months Ago

Google is hoping other Search Engines also make use of this protocol.

Technically the sitemap belongs to YOU. Share it with anyone you like.

I am hoping the protocol expands to include TITLE, DESCRIPTION, and KEYWORD meta tags. Think of how convenient it would be to have this info summarized.

Then with any luck major directories (DMOZ) could automate their submit process as well. :lol:
  • mighty b
  • Beginner
  • Beginner
  • mighty b
  • Posts: 39

Post 3+ Months Ago

Quote:

According to Google OK means that the uploaded file processed correctly and is now queued. Of course there is no guarantee the "suggestions" will be included in the index. :lol:


Il have to wait for a few weeks then lol
  • pine_things
  • Novice
  • Novice
  • pine_things
  • Posts: 20
  • Loc: Warwickshire, UK

Post 3+ Months Ago

has anybody experienced better indexing of their webpages by using the google site map?

does it see all the pages quicker?
  • mighty b
  • Beginner
  • Beginner
  • mighty b
  • Posts: 39

Post 3+ Months Ago

Defenetely. Google bot lives on my forum now. I had quite a few pages indexed the other day that google bot would not even go to before.
  • 993ti
  • Newbie
  • Newbie
  • 993ti
  • Posts: 12

Post 3+ Months Ago

Submitted it and it got downloaded within the hour, that's pretty fast.
I'm curious how it goes :)
  • Jibran
  • Beginner
  • Beginner
  • User avatar
  • Posts: 61
  • Loc: Lucknow, India

Post 3+ Months Ago

I just noticed that the number of indexed pages of my site have gone down from 5000 to 4870! 8O This happened after using the Google Sitemaps.
  • mighty b
  • Beginner
  • Beginner
  • mighty b
  • Posts: 39

Post 3+ Months Ago

:bouncingsmile: result!!!

Just got 4000+ pages indexed. Before the site map it took me 5 months just to get 300
  • joebert
  • Fart Bubbles
  • Genius
  • User avatar
  • Posts: 13502
  • Loc: Florida

Post 3+ Months Ago

Somthing for thoose of you with phpBB boards & trouble getting them indexed may want to keep an eye on, http://www.phpbb.com/phpBB/viewtopic.php?t=296051
  • Johan007
  • Guru
  • Guru
  • User avatar
  • Posts: 1080
  • Loc: Aldershot, UK

Post 3+ Months Ago

This really should be good news for webmasters with content and news sites. Imagine your articles going into Google within hours instead of days! My entire site map for Future Movies is only 50kb and sure it has its limitations but those are insignificant.

You can just provide a list of URL’s in XML if you want to be safe however I also strongly recommend anyone using a database to avoid the last update date because that date is taken from the database and if you make changes to the HTML your page maybe ignored.

Freel free to use my simple classic ASP code (easy to convert to PHP). To save on file size do not convert this code to inline code (its worth the slight server hit for multiple "response.write" becuase only Google will be using it):

Code: [ Select ]
<!-- #Include virtual="/database-connection.asp" -->
<%

Response.Buffer = true
response.ContentType = "text/xml"
response.write "<?xml version='1.0' encoding='UTF-8'?>"
response.write "<urlset xmlns='http://www.google.com/schemas/sitemap/0.84'>"

' List your static URL's

response.write "<url>"
response.write "<loc>http://www.domain.co.uk/</loc>"
'response.write "<lastmod>" & Danger & "</lastmod>"
response.write "<priority>0.5</priority>"
response.write "<changefreq>daily</changefreq>"
response.write "</url>"

response.write "<url>"
response.write "<loc>http://www.domain.co.uk/sub-home</loc>"
'response.write "<lastmod>" & Danger & "</lastmod>"
response.write "<priority>0.5</priority>"
response.write "<changefreq>daily</changefreq>"
response.write "</url>"

' List your dynamic URL's

Dim n

n = 0

strSql = "SELECT *, Table.ID AS [pageID] WHERE Table.Delete<>'Y' ORDER BY Created DESC"
Set db = Server.CreateObject("ADODB.Connection")
Set Rs = Server.CreateObject("ADODB.Recordset")
db.Open strDBConnection
Rs.Open strSql, db

Do While Not Rs.EOF
    intID = Rs.Fields("pageID").Value
    response.write "<url>"
    response.write "<loc>http://www.domain.co.uk/article.asp?ID=" & intID & "</loc>"
    'response.write "<lastmod>" & Danger & "</lastmod>"
    
    If n < 10 Then
        response.write "<priority>1.0</priority>"
        response.write "<changefreq>daily</changefreq>"
    ElseIf n < 20 Then
        response.write "<priority>0.7</priority>"
        response.write "<changefreq>monthly</changefreq>"
    Else
        response.write "<priority>0.2</priority>"
        response.write "<changefreq>yearly</changefreq>"
    End If
    
    n = n + 1
    
    response.write "</url>"
    Rs.MoveNext
Loop

Rs.Close
db.Close
Set Rs = Nothing
Set db = Nothing

' End Dynamic URL (maybe have another table?)

response.write "</urlset>"
%>
  1. <!-- #Include virtual="/database-connection.asp" -->
  2. <%
  3. Response.Buffer = true
  4. response.ContentType = "text/xml"
  5. response.write "<?xml version='1.0' encoding='UTF-8'?>"
  6. response.write "<urlset xmlns='http://www.google.com/schemas/sitemap/0.84'>"
  7. ' List your static URL's
  8. response.write "<url>"
  9. response.write "<loc>http://www.domain.co.uk/</loc>"
  10. 'response.write "<lastmod>" & Danger & "</lastmod>"
  11. response.write "<priority>0.5</priority>"
  12. response.write "<changefreq>daily</changefreq>"
  13. response.write "</url>"
  14. response.write "<url>"
  15. response.write "<loc>http://www.domain.co.uk/sub-home</loc>"
  16. 'response.write "<lastmod>" & Danger & "</lastmod>"
  17. response.write "<priority>0.5</priority>"
  18. response.write "<changefreq>daily</changefreq>"
  19. response.write "</url>"
  20. ' List your dynamic URL's
  21. Dim n
  22. n = 0
  23. strSql = "SELECT *, Table.ID AS [pageID] WHERE Table.Delete<>'Y' ORDER BY Created DESC"
  24. Set db = Server.CreateObject("ADODB.Connection")
  25. Set Rs = Server.CreateObject("ADODB.Recordset")
  26. db.Open strDBConnection
  27. Rs.Open strSql, db
  28. Do While Not Rs.EOF
  29.     intID = Rs.Fields("pageID").Value
  30.     response.write "<url>"
  31.     response.write "<loc>http://www.domain.co.uk/article.asp?ID=" & intID & "</loc>"
  32.     'response.write "<lastmod>" & Danger & "</lastmod>"
  33.     
  34.     If n < 10 Then
  35.         response.write "<priority>1.0</priority>"
  36.         response.write "<changefreq>daily</changefreq>"
  37.     ElseIf n < 20 Then
  38.         response.write "<priority>0.7</priority>"
  39.         response.write "<changefreq>monthly</changefreq>"
  40.     Else
  41.         response.write "<priority>0.2</priority>"
  42.         response.write "<changefreq>yearly</changefreq>"
  43.     End If
  44.     
  45.     n = n + 1
  46.     
  47.     response.write "</url>"
  48.     Rs.MoveNext
  49. Loop
  50. Rs.Close
  51. db.Close
  52. Set Rs = Nothing
  53. Set db = Nothing
  54. ' End Dynamic URL (maybe have another table?)
  55. response.write "</urlset>"
  56. %>


lastmod, priority and changefreq are all optional tags! Its unlikly they will be used but if they are then don’t forget its all relative so do try and have low values for all you’re old pages, I suggest priority for homepage 0.5 old pages 0.2 and new pages 1.0 which you can do dynamically. Same goes for changefreq.

Limitations:
This code obviously does not limit the URL count to 50,000 so keep an eye on that if you have a mega site – maybe add another counter to show number of URL in admin mode. The way it’s coded (non inline) the file size would never get to 10MB for 50,000 url’s and more like 5MB.
  • pine_things
  • Novice
  • Novice
  • pine_things
  • Posts: 20
  • Loc: Warwickshire, UK

Post 3+ Months Ago

Jibran wrote:
Follow my advice dudes! Dont jump Sitemaps! My pages indexed had reached 5000, now only 724 are left, in four days' time!!! The decrease started occuring after I had started using Sitemaps!!! :angry:


the header of all your pages looks pretty similar. perhaps google thinks that there are similar pages. before it could not go deep down and look at the content. now that you have created the sitemap it looks at all your pages. also the priority in your sitemap.xml for the majority of the url is 1.0.
  • Johan007
  • Guru
  • Guru
  • User avatar
  • Posts: 1080
  • Loc: Aldershot, UK

Post 3+ Months Ago

How many pages are listed in your sitemap?
  • Jibran
  • Beginner
  • Beginner
  • User avatar
  • Posts: 61
  • Loc: Lucknow, India

Post 3+ Months Ago

1200+ pages are listed. The priority has been set automatically, I'll change that too, i.e. 8 pages and 4 list of urls.txt are listed in the config.xml this when run under sitemap_gen.py gives 1210 html and 1 php page.

Also the titles for almost every page is different. I have set up dynamic titles. The titles and descriptions for every news article is different etc.
  • Johan007
  • Guru
  • Guru
  • User avatar
  • Posts: 1080
  • Loc: Aldershot, UK

Post 3+ Months Ago

Jibran if your talking about http://www.pottersrealm.com sadly I suspect you havent got enough PR to support that many pages. PR3 is no way enough! Make a links paage and start swapping with other potters. You need to aim for PR5 maybe 6.
  • Jibran
  • Beginner
  • Beginner
  • User avatar
  • Posts: 61
  • Loc: Lucknow, India

Post 3+ Months Ago

Does PR matter with the no. of pages being indexed? The following: http://www.searchengineengine.com/
shows my real PR at 4. I have added my site to some directories too. I am targetting for a higher PR. And I do have a links system:

http://www.pottersrealm.com/links.html

Does it matter to whom I link too? or only who links me?

Edit: just a question, could it be that since other high PR sites have similar content to my site, that my site is getting penalised? I did a search via Copyscape.com and found quite an overwhelming number of matches!
  • Johan007
  • Guru
  • Guru
  • User avatar
  • Posts: 1080
  • Loc: Aldershot, UK

Post 3+ Months Ago

Copyscape does not show up only similar content but more like theft content then its a possibility but this latest Bourbon update by Google is unknown. If you search for "Potters Realm Harry Potter!" you will not be number one unless you remove the Google penilisation filter and it then shows you number one. Have your stats fallen this month (or from the 24th last month?)

If yes to both then:

1. Remove dupe content
2. Get a few more inbound links from other potter sites.

We are going way offtopic here... start a new post or PM if needed.
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

Of course they're going to be good. They allow you to easily say to Google "Hey, these pages exist on my site".

Even if they're several links deep into your content, you can tell Google that they are around on your site and available for public viewing.

You'll be able to get a lot more of your pages into Google's indexing queue much quicker than natural crawl (at least, that's the theory).

It doesn't replace natural crawl, it's a compliment to it.

I use it on some of my sites.
  • boohlick
  • Beginner
  • Beginner
  • User avatar
  • Posts: 61

Post 3+ Months Ago

Thats right... but you have to make it in xml form to be able to add it in your google site map.. Its good, like what he says.. google can easily crawl all your pages..
  • SplitMedia
  • Newbie
  • Newbie
  • SplitMedia
  • Posts: 13
  • Loc: Split-Media.com

Post 3+ Months Ago

For http://www.split-media.com

How would I get this "xml" what information needs to be on it? I didn't understand it very well, and I signed up with this google sitemap.. Not really understanding what was involved in it.. Any comments/suggestions?
  • malprave
  • Student
  • Student
  • malprave
  • Posts: 84

Post 3+ Months Ago

I have just completd my XML sitemap and submitted it to Google (not really, they are so busy I have to try in a few moments).

I did not, however, do it the way they perscribed (Using Python and all that), I just googled some things and found this website:

http://www.auditmypc.com/free-sitemap-generator.asp#sitemap-generator-updates

which does exactly the same thing except you don't need Python, all you need is a Java enabled browser.

It can take a while to change all the setting if you want to customise each page i.e. priority, modification occurance...

I just wanted to know how many others have done this and if we get any additional benefits!!!???
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

I wrote my own Google Sitemaps generators for my sites, and for several popular PHP based scripts.

Submitted, and traffic has increased to those sites beyond their normal expected growth.
  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Posts: 23456
  • Loc: Woodbridge VA

Post 3+ Months Ago

Axe provided me with his script for phpBB forums, and it is working for me as well. His version is available on various download sites.

Here's one of them
http://www.hotscripts.com/Detailed/50901.html
  • reaper
  • Proficient
  • Proficient
  • User avatar
  • Posts: 435
  • Loc: europe

Post 3+ Months Ago

I came accros this site and it has a nice free tool for generating a google sitemap that you can submit.

You should check it out

http://www.vigos.com/products/gsitemap/

Note:
Microsoft .NET Framework is required to use Gsitemap!
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

no manual, no FAQ, no other documentation (don't know how that helps beginners, heh).

I'll stick to writing my own. If you've got enough pages that you can really justify having a sitemap, I think a remote piece of software, or remote script like that would just put too much of a load on the server loading all the URLs on the site.
  • Alan Lastufka
  • Proficient
  • Proficient
  • User avatar
  • Posts: 318
  • Loc: ChicagoLand, IL, USA

Post 3+ Months Ago

http://www.sitemapbuilder.net/default.aspx

That's gotta be the easiest for beginners or anyone really. You can tell it to analyze your server - or just copy and paste a link list form a word doc (I keep a text file I add every new link to myself) and it spits out an .xml file.

Super easy.
  • funlounge
  • Beginner
  • Beginner
  • funlounge
  • Posts: 40

Post 3+ Months Ago

Hi
I'm using a sitemap.xml file (submitted to Google sitemaps)
for my new site www.celebritiescentral.net
a week ago now, but still my page does not appear in its index

Any experiences with sitemaps ?

Thanks
  • Axe
  • Genius
  • Genius
  • User avatar
  • Posts: 5739
  • Loc: Sub-level 28

Post 3+ Months Ago

The sitemaps file just gets your URLs in Google's database faster, it doesn't make them take any less time to actually view your pages. They'll put your URLs in their spider queues, and if they come across backlinks, etc. they'll bump URLs up the queue and go from there...

So, it still takes as long as conventional ways to START getting indexed, although with Sitemaps, once it does start getting indexed, it'll ALL start getting indexed, so it's much faster overall, especially once Google's caught up and you're just adding new content.
  • mac5150
  • Novice
  • Novice
  • mac5150
  • Posts: 15
  • Loc: Colorado

Post 3+ Months Ago

Google give some good links to both downloadable and online site map generators.
http://code.google.com/sm_thirdparty.html

I'm using GsiteCrawler right now. It does what I need and it's pretty stable for a beta.
  • LittleEarner
  • Novice
  • Novice
  • LittleEarner
  • Posts: 18
  • Loc: England, UK

Post 3+ Months Ago

I've just tried GsiteCrawler based on your recommendation, and I must say, I'm very impressed with it. I'd certainly recommend it.

Dan
  • Anonymous
  • Bot
  • No Avatar
  • Posts: ?
  • Loc: Ozzuland
  • Status: Online

Post 3+ Months Ago

Post Information

  • Total Posts in this topic: 106 posts
  • Users browsing this forum: No registered users and 5 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.