Knowing how to read, and follow, sitemap entries is a useful skill in diagnosing many Blogger problems.
The Blogger generated sitemaps, which apply to each blog, index both pages and posts. This blog, like every other blog, has two sitemaps, automatically generated.
"sitemap.xml" (posts) roughly duplicates the "Archives" gadget - but has advantages, over the gadget. "sitemap-pages.xml" (static pages) is not redundant, to any Blogger accessory.
The two Blogger generated sitemaps provide possibilities for content and problem analysis, unequaled by any Blogger accessory.
The posts sitemap is valuable.
- The Archives gadget, which provides a canonical list of posts, is not included on all blogs.
- The Archives gadget, for most blogs, must be examined one month at a time - and does not provide a search option.
- Manually searching for a post, using "Older Posts", is time consuming.
The pages sitemap is especially valuable.
- There is no canonical accessory, that lists pages.
- The Pages gadget only lists pages that the owner wants to index.
You can view pages, and posts, using either sitemap. Note only a published page or post will be listed. Draft status pages and posts will not be listed.
This blog, like many, has a paged posts sitemap.
Each posts sitemap page will list up to 150 entries. Any blog with over 150 posts will have a paged sitemap.
http://blogging.nitecruzr.net/sitemap.xml
The posts sitemap, for this blog.
Page 1 of the posts sitemap contains the 150 most current posts.
http://blogging.nitecruzr.net/sitemap.xml?page=1
Page 1 of the posts sitemap, for this blog. It's a very simple display - just URL and "last edited" date / time, for each post.
Here is the pages sitemap.
http://blogging.nitecruzr.net/sitemap-pages.xml
I don't have a lot of pages published, in this blog, that are of public interest.
You can access pages and posts, from the sitemap.
Using browser features, you can access any listed page or post, directly from either display.
Highlight the URL that you need to examine.
Right click, select "Go to http://blogging.nitecruzr.net /p/what-are ...".
And there is the page in question.
You can access "http://blogging.nitecruzr.net/p/what-are-differences-between-pages-and.html" from the pages sitemap - or "http://blogging.nitecruzr.net/2013/12/what-are-differences-between-pages-and.html" from the posts sitemap.
Most browsers have a right click context menu, and a "Go to (URL)" selection - and provide direct access to any page or post, using a properly formatted URL.
One practical case, diagnosed using the sitemaps.
One of the intriguing uses of the sitemaps, that we find useful, involves a blog with content - yet when viewing it, we see
No posts.
in the blog "status message" section. Why do I see "No posts." - instead of blog content?
People who confuse "pages" (aka "static pages") and posts (aka "dynamic pages") may publish a blog, using nothing but static pages. The blog main page will normally only display posts - and show "No posts.", when viewed.
You can, however, redirect the home page to a given static page - and add links between the static pages. Or, republish the pages as posts, if convenient.
The recently added #Blogger generated pages and posts sitemaps offer canonical access to all published pages and posts, in every publicly accessible blog. These sitemaps are useful to people - as well as to search engines and other indexing processes.
Last week, Blogger gave us a feature that various blog owners have asked about, for many years.
The previous sitemap, based on the blog posts feed, has been replaced by an automatically generated, dedicated sitemap. You can see one, for this blog, as an example.
Accompanying the new sitemap will be an updated "robots.txt" file.
The new sitemap will be very simple.
http://blogging.nitecruzr.net/sitemap.xml
The sitemap will include 2 data elements / post.
- Post URL
- Last updated date / time (UTC).
The new sitemaps offer interesting diagnostic possibilities, for various blog problems.
By eliminating the posts newsfeed, sitemap access becomes much cleaner.
With these data elements now available without requiring searching through the post content in the newsfeed, any process which indexes or searches, using any of these data elements, will be much simpler - and be more stable, when run.
My suspicion is that several Blogger / Google features, no longer immediately requiring the blog feed in indexing, will be much more usable. Blogs which use dynamic templates, the Reading List, and search engine indexing, will eventually benefit.
Accompanying the new sitemap, which will index posts, will be a sitemap for static pages. You can see a pages sitemap, for this blog, as an example.
http://blogging.nitecruzr.net/sitemap-pages.xml
The pages sitemap appears to have 2 data elements / static page.
- Page URL
- Published date / time (UTC).
You will see the new sitemap specified in the "robots.txt" file.
Check the "robots.txt" file on your blog. When the sitemap is installed on your blog, you will see the change.
If you're unfamiliar with the concept, you may read my other posts in this blog - or the Webmaster Tools Help: Learn about sitemaps. Now, we can do other things with the blog feed, without impeding indexing. Possibly, even private blogs can now be indexed.
Large sitemaps will be broken into pages.
Any sitemap with over 150 entries (pages or posts) will be broken into pages - 150 entries / sitemap page, automatically.
Examine the posts sitemap, for this blog - as of 5 June, 2016.
http://blogging.nitecruzr.net/sitemap.xml
<?xml version='1.0' encoding='UTF-8'?><sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=1</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=2</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=3</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=4</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=5</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=6</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=7</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=8</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=9</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=10</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=11</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=12</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=13</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=14</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=15</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=16</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=17</loc></sitemap><sitemap><loc>http://blogging.nitecruzr.net/sitemap.xml?page=18</loc></sitemap></sitemapindex>
The most current posts will be listed on Page 1.
http://blogging.nitecruzr.net/sitemap.xml?page=1
The sitemap will have a limited size.
The sitemap will provide a maximum of 3,000 entries - 20 pages at 150 posts / page. As new posts are published to the blog, they will be added, automatically. Hopefully, not too many blogs will have 3,000 posts published, before the blog is indexed.
Since the announcement was made, I have added maybe a dozen posts to this blog. I just looked at Page 1 of the sitemap for this blog, and this post is now, there - 5 minutes after this post was published. You may, or may not, see the same update promptness on your blog.
The old sitemap is now not needed.
Both the old and new sitemaps index the same post complement - the old "sitemap" (posts feed) simply contains irrelevant content - the post material.
Let's compare the old sitemap, with the new, using either the content itself or an HTTP trace pair. Click on two of the links below, and compare the results.
The old sitemap:
The old sitemap URL:
http://blogging.nitecruzr.net/feeds/posts/default?redirect=false
The old sitemap HTTP trace:
http://www.rexswain.com/cgi-bin/httpview.cgi?url=http://blogging.nitecruzr.net/feeds/posts/default%3Fredirect%3Dfalse&uag=Mozilla/5.0+(X11%3B+CrOS+armv7l+7978.74.0)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/50.0.2661.103+Safari/537.36&ref=http://www.rexswain.com/httpview.html&aen=&req=GET&ver=1.1&fmt=TXT
The new sitemap:
The new sitemap URL:
http://blogging.nitecruzr.net/sitemap.xml
The new sitemap HTTP trace:
http://www.rexswain.com/cgi-bin/httpview.cgi?url=http://blogging.nitecruzr.net/sitemap.xml&uag=Mozilla/5.0+(X11%3B+CrOS+armv7l+7978.74.0)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/50.0.2661.103+Safari/537.36&ref=http://www.rexswain.com/httpview.html&aen=&req=GET&ver=1.1&fmt=TXT
The old and new sitemaps index the same content. The old sitemap simply includes all of the post content, as blog feed - and the new sitemap includes only search engine useful data. Some of the processes that read sitemaps will simply be able to digest the new sitemap easier - they simply walk the sitemap, to index the posts themselves.
Last week, Blogger gave us a feature that various blog owners have asked about, for many years.
The current sitemap, based on the blog posts feed, is being replaced by an automatically generated, dedicated sitemap. You can see one, for this blog, as an example.
Accompanying the new sitemap will be an updated "robots.txt" file.
The new sitemap is not being setup, immediately, on all blogs. Only blogs with standard "robots.txt" file will get the sitemap, initially. It's being installed, automatically, with no action required by the blog owner, on a limited number of blogs.
I've seen a handful of blog owners report seeing the new sitemap being installed, on their blogs.
The sitemap will include 3 data elements / post.- Post Title.
- Post URL
- Published date / time (UTC).
With these data elements now available without requiring searching through the post content in the newsfeed, any process which indexes or searches, using any of these data elements, will be much simpler - and be more stable, when run.
My suspicion is that several Blogger / Google features, no longer immediately requiring the blog feed in indexing, will be much more usable. Blogs which use dynamic templates, the Reading List, and search engine indexing, will eventually benefit.
Accompanying the new sitemap, which will index posts, will be a sitemap for static pages. You can see a pages sitemap, for this blog, as an example.
The pages sitemap appears to have 2 data elements / static page.- Page URL
- Published date / time (UTC).
Check the "robots.txt" file on your blog. When the sitemap is installed on your blog, you will see the change.
If you're unfamiliar with the concept, you may read my other posts in this blog - or the Webmaster Tools Help: Learn about sitemaps. Now, we can do other things with the blog feed, without impeding indexing. Possibly, even private blogs can now be indexed.
The sitemap will provide a maximum of 2,500 entries - 5 pages of 500 entries, each page. As new posts are published to the blog, they will be added, automatically. Hopefully, not too many blogs will have 2,500 posts published, before the blog is indexed.
Since the announcement was made, I have added maybe a dozen posts to this blog. I just looked at Page 1 of the sitemap for this blog, and this post is now, there - 5 minutes after this post was published. You may, or may not, see the same update promptness on your blog.
Some Blogger blog owners don't know how to setup a sitemap, for their blog.
Setting up the sitemap, for a website, is a major process - and takes time. Every time you add a page to a website, the sitemap has to be updated - or how do your readers find the new page?
Alternatively, you can use a sitemap builder service, which builds the sitemap. This gives you a file, hosted by the sitemap builder service. Will the sitemap builder service be in business next week?
In either case, you take your sitemap file, and upload it to the blog. Every time you add a page, you update the sitemap, then you upload the updated sitemap. Every time - or the new page remains unindexed, until you do.
(Update 2014/11/04): Blogger is now providing an automatically generated, dedicated sitemap, to replace the newsfeed sourced sitemap - and an automatic "robots.txt" update. There is no need for a custom sitemap - nor to update "robots.txt".
With a Blogger blog, you designate the sitemap, using Google Webmaster Tools, referencing the URL of the posts newsfeed.
You can substitute any custom sitemap, if you wish, but this will be at your own risk - and you can develop your own installation instructions.
Once done with the Webmaster Tools sitemap wizard, you are free to work on the visible parts of the blog. Every time you publish a new post, the post is updated into the newsfeed, and becomes part of the sitemap.
The next time a search engine bot hits the blog, it picks up the new sitemap - and indexes the new post.
Combine the automatically and immediately updated sitemap, with more time spent updating the blog, and you get happier readers, and better search engine reputation. That's a win - win.
If your blog gets very large, you can add a new sitemap segment - at your convenience - for every 500 posts.
Why spend time manually updating a sitemap, with every new post (and worry about a file, hosted by someone who may or may not stay in business)? Use the posts newsfeed (hosted by Google), and work on the blog content.
For a Blogger blog, proper indexing by the search engines is critical to the success of the blog, in getting readers.
Some blog owners are disappointed to find that their blog has no page rank, and no visibility in the search engine results - and little to no chances for getting readers.
Not all blog owners understand details about the custom domain migration process, any issues related to renaming the blog, or simply how to get a blog properly indexed.
One of the most useful tools, that you as a blog owner can use, is the sitemap.
With a Blogger blog, that is typically a collection of posts indexed by date and maybe by topic, the sitemap, properly presented to the search engines, helps the search engines index all of the posts methodically.Submit a Sitemap to tell Google about pages on your site we might not otherwise discover.
That's the simple advice, from Google, which you should see.
When I created this blog, and submitted my first sitemap, Google setup a default.http://bloggerstatusforreal.blogspot.com/feeds/posts/default?orderby=updated
Had you looked at the "robots.txt" file, when this blog was published as "bloggerstatusforreal.blogspot.com", that's what you would have seen.
Having later published this blog as "blogging.nitecruzr.net", Google setup a new default sitemap.http://blogging.nitecruzr.net/feeds/posts/default?orderby=updated
You may look at my current "robots.txt" file, and that's what you will see.
The default sitemap indexes the most recent 26 posts. For most blogs, especially new ones, you will seldom publish more than that number between each indexing pass by the search engines - and a 26 posts submission is generally sufficient.
For a blog that's been published for a while, and has more than 26 posts, you will need a more robust sitemap than the default. When you change the URL of the blog, or when you use Jump Break on main page posts, you need a sitemap complement that indexes each individual post, one by one.
Note that any sitemap will be much more useful, with the blog publishing a "Full" feed.
In the example of this blog, which now has over 1,500 posts, I would add 4 sitemaps (each sitemap to submit a maximum of 500 posts)- http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1&max-results=500
- http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=501&max-results=500
- http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1001&max-results=500
- http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1501&max-results=500
Having added the right complement of sitemaps, you watch the display in the Sitemaps wizard:
Submitted URLs
1,539
1,535 URLs in web index
Having submitted those sitemaps last week, 1,535 out of 1,539 posts, allowing for some search engine indexing latency, is about right.
Your blog, having a different number of posts, may use a different complement of sitemaps - but you can use a similar strategy, in determining the number to use. And however you look at it, adding a new sitemap segment for every 500 posts beats manually updating the sitemap for every new post.
>> Top
For a Blogger blog which needs a sitemap for indexing, changing the URL requires creating a new sitemap.
For a blog newly published to a different URL - either a non BlogSpot URL, using custom domain publishing, or simply a different BlogSpot URL - the first few days under the new URL are critical to the success of the blog. Some blog owners find, to their horror, that their blog suddenly has no page rank, and has dropped out of visibility in the search engine results.I bought a domain from Google, and my blog is now invisible. Obviously, this is another swindle by Google, stealing my money!
These blog owners have not done any research, about the custom domain migration process, or about any issues related to renaming the blog. The successful migration process starts with getting the blog, under its new URL, indexed by the search engines.
One of the most useful tools, that you as a blog owner can use, is a sitemap, now provided automatically, by Blogger.
The new automatic sitemap will index the most recent 2,500 posts. For most blogs, that will be sufficient.
When you re publish your blog under a new URL, you want all posts reindexed promptly. To get all posts reindexed, if your blog contains more than 2,500 posts, you are going to need a more robust sitemap complement.
A sitemap, to support a blog with more than 2,500 posts, will require new entries, in Webmaster Tools.
You want your blog, under the new URL, to be visible under that new URL as soon as possible, so setup a proper sitemap, if necessary - then get to work publishing new posts, and increase search engine activity. And, use Jump Break for all new posts, to focus search engine attention on the individual posts.
When we rename a blog, we use the Post Feed Redirect URL to redirect all feed references, from the old URL to the new feed URL.
When we setup a custom feed for a blog, maybe using FeedBurner, we use the Post Feed Redirect URL to redirect all feed references, from the blog, to the FeedBurner feed. That lets us use a FeedBurner feed without having to change all of the feed links in the blog.
Besides using the Post Feed Redirect URL to setup a FeedBurner feed, you can use the Post Feed Redirect URL when you re publish the blog to a new BlogSpot URL.
Please! Be sure to setup the Post Feed Redirect in the proper direction.
Using the post feed redirect is a simple, 1 - 2 - 3 exercise.
- Publish a stub blog, to the old URL.
- Determine the URL of the blog post feed, under the new URL.
- Set the post feed redirect, for the stub blog.
Publish a stub blog, to the old URL.
Immediately after you re publish the blog to its new URL, you should be setting up a stub blog, to visually point your readers to the new URL. Besides visually pointing your readers, you can re point your subscribers - again using the stub blog.
Determine the URL of the blog post feed, under the new URL.
Copy the updated feed URL from the blog header. Precision is essential when redirecting the feed, so let Blogger do the work for you here - determine the new post feed URL, properly.
Set the post feed redirect, for the stub blog.
Go to Settings - Other, for the stub blog, published to the old URL. Edit the "Post Feed Redirect URL", and paste the new feed URL, as just copied.
When you do this, all subscribers to your blog, under the old URL, should see the blog feed in its new home, with linked content. The search engines, indexing the blog using a sitemap based upon the old URL, should start indexing it using the new URL.
Properly done, this should give your blog, under the new URL, a good start towards regaining the search engine reputation that it had, under the old URL.
When we setup a Blogger blog, we give up control of files and folders, and let Blogger maintain the structure of the blog.
When we publish a Blogger blog (either to BlogSpot, or to a Google Custom Domain), all that we do is post, and maintain the blog template. Even with the files and folders controlled by Blogger, and normally hidden from view, there are ways to examine the contents of some files.
Those who would be better off not examining file content do so anyway, become confused, and stress themselves needlessly. Every week, we read anxious queriesHelp! My blog has been hacked!!
orMy robots.txt file is blocking my blog from being indexed!
You can make two Settings changes that are relevant here, but mostly this file is maintained by Blogger code.
Blogger maintains "robots.txt", in each blog, on our behalf.
Occasionally, Blogger makes changes to our blogs in general, and changes the content of "robots.txt" to support the changes made. Recently, changes to "feedReaderJson" necessitated a change to "robots.txt".
Here's the "robots.txt" file for this blog, "blogging.nitecruzr.net", as of 2016/01.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://blogging.nitecruzr.net/sitemap.xml
There are 3 main entries, in a standard "robots.txt" file.
Here we see 3 entries. For an explanation of some terms, and for a demonstration illustrating the results, please refer to Google Webmaster Tools - "Analyze robots.txt". You may also be enlightened by reading The Web Robots Pages.
- This allows access to all blog components of the blog ("Disallow: (null)"), to the spider "Mediapartners-Google", which is the spider that crawls pages to determine AdSense content. This entry overrides the following entry, for the specified spider.
User-agent: Mediapartners-Google
Disallow:
- This disallows indexing from all URLs containing "/search"(ie, label searches) - and it allows all other blog URLs.
User-agent: *
Disallow: /search
Allow: /
- This defines the URL of the sitemap.
Sitemap: http://blogging.nitecruzr.net/sitemap.xml
The sitemap is separate from the blog feed, and automatically provided.
You can use Search Console, for an analysis of your "robots.txt" file.
Here's my analysis of "robots.txt" for this blog, run using the wizard. You'll want to view this in full screen mode, zoomed as highly as possible.
Note the hypothetical effects of indexing 3 hypothetical blog URLs, against both "Googlebot" and "Mediapartners-Google", shown at the very bottom.
I've now examined a dozen or so different similar files, for blogs published to both native BlogSpot and custom domains, and excepting the URL of the sitemap, all files have been identical in content. My conclusion is that this is a normal file, and unless we start seeing a flood of complaints about indexing problems, I see no reason to suspect a problem.
So, the next time someone comes to you moaning
My robots.txt file is blocking my blog from being indexed!
you can assure them
No, your "robots.txt" file is normal.
Then, introduce them to Google Webmaster Tools, and its many diagnostic reports.
As a Google customer, we have access to a very useful set of tools, which help us develop and maintain a healthy relationship between our blogs and the Google search engine spiders.
Google Webmaster Tools (now Search Console) provides us with tools to measure our visibility, and it provides us with helpful hints on how to improve our visibility.
You can use Google Webmaster Tools for any active URL that you administer.
If your blog is published to BlogSpot, you use the "xxxxxxx.blogspot.com" URL; if the blog is published to a custom domain, you use the primary URL for the domain.
Using Google Webmaster Tools for any web site requires that you are the administrator of the web site.
Google doesn't want anybody who's not the administrator, of any web site, accessing search engine diagnostics for that web site.
The meta tag provided in the verification process is a token, and it must match the URL that you are verifying. If you change the active URL for the blog (rename it within "blogspot.com", or publish it to a custom domain), you will need a new Google Webmaster Tools entry, with a new verification tag.
The meta tag is as important a data element as the clickable link that you get in your email, when you're offered membership in a blog - or when you verify an email address, for an account. You click on the link, which opens the token, and applies it to your account.
With a Blogger blog in Google Webmaster Tools, you install the token into the blog template, to verify ownership of the domain (blog). Only a blog administrator has the authority to update the template. If you don't install the token into the template, you can't prove that you own the blog, and you can't use Google Webmaster Tools for your blog.
Elsewhere in Google Webmaster Tools, you'll find the Web crawl logs. Some information in the web crawl logs can be very useful to you, while other is informational only. It's to your advantage to separate the two categories. The "Preferred domain" setting in "Site Configuration" is a very small setting, that can have a big impact. And most of us setup our blogs on GWT so we can add a sitemap to the blog.
>> Top
Custom feed services, like FeedBurner, give you much more than a native Blogger blog feed. You can aggregate content and pull in material from other blogs and web sites, you can provide more user friendly formatting, and you can even track the number of readers who subscribe to a FeedBurner custom feed.
Your Blogger blog even supports your use of a custom feed. If you setup the Post Feed Redirect option, all links in the blog that would reference the native blog post feed are replaced by the custom feed. Your blog readers, when they click on a link that saysClick here to subscribe to my feed!
automatically subscribe to the custom feed. This is good for your readers, and for you.
Unfortunately, use of the feed redirect option isn't a good idea for everybody, immediately. If you're using a Google Sitemap, to make it easier for the search engines to index your blog, and bring you more readers, you'll need to use the native blog feed.
With the blog redirecting all blog post feed references, to a custom feed, the sitemap will be broken. The Google Sitemap requires a feed that has the same URL as the blog. A custom feed has the URL of the custom feed service. This breaks the sitemap, unless you modify the feed URL.
Having modified my feed URL, I can create a Google sitemap that the spider can work from.
>> Top
