Make Your Blog Searchable - Where To Start

Making a blog searchable is the first step in getting search traffic, and readers, for our blogs - when we want them visible to the public.

Not all blog owners want their blogs visible to everybody. Complementing the ability to publish a blog, and have it visible only to invited readers, Blogger gives us the ability to make the blog invisible, to the search engines.

If we want readers for a blog, and do not have a private reader audience in mind, we first make it visible to the search engines.

The Privacy wizard, on the Blogger dashboard, lets us choose whether to make a blog visible to the search engines.

Start with the dashboard Settings - Basic page, and Edit the Privacy setting.

Select "Yes" for both options - if you want the blog indexed.



Select "Yes" for both options, if you want the blog indexed. This is how you start getting search traffic, and readers, for your blog.

• Add your blog to our listings?
• Let search engines find your blog?

And click "Save changes".

Having made the blog visible to the search engines, continue by publishing lots of informative, interesting, and unique content. Make it a blog that people will read - and that search engines will index.

If you prefer the blog to be invisible or private, on the other hand, select "No" for both options. Then continue by developing your private audience, as you like.

There are more settings - for owners who want to fine tune search engine access. Both the Blogger dashboard Settings - "Search preferences" page, and the "Google Search console" (previously known as "Webmaster Tools"), have a rich collection of settings, complementing "Privacy".

If you really want to fine tune these settings, look at the Settings - "Search preferences" page, and the "Crawlers and indexing" section - but read and heed the serious advice.

Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.

You can read more advice in Blogger: Help people find your blog on search engines - and Webmasters: Search Engine Optimization (SEO) Starter Guide. And you'll find Webmaster Central: Crawling, Indexing, and Ranking and Webmaster Central: Search Console useful for personal advice, complementing Blogger Help Forum.

Since Webmaster Forums supports owners of static websites, as well as owners of Blogger blogs (and other blogs), not all Search Console / Webmaster advice and tools will be useful for Blogger blogs. Be selective when considering Webmaster instructions - and ask questions in Blogger Help Forum, any time you feel the need.

The "Privacy" wizard is where you start - and similarly where some blog owners should stop.

---

The Blogger "Privacy" wizard, on the "Basic" Settings page, provides the start to getting a blog indexed by the search engines - when we want a blog indexed. Not all blog owners want their blog indexed - and that need makes the "Privacy" settings relevant.

Static Blogs, And Spam Classification

Blogger blogs are starting to become a regular website publishing platform - not just a blogging platform.

Long ago, a Blogger blog was just an online journal, with posts, and text. Then, Blogger added labels, later static pages - and then the ability to replace the home / main page, with dynamic or static content of your choice. Recently, the original limitation of 20 static pages / blog was relaxed (we are not yet certain that the limit has been eliminated completely, however).

Now, you can practically publish a static website, without even having posts, using a Blogger platform. But, is this a good idea, with Blogger "One button" Publishing?

We have people asking various questions, in Blogger Help Forum: Learn More About Blogger, about making a static website.

How do I remove posts from my main page?

and

How do I publish a blog, without posts?

And alternately, in Blogger Help Forum: Get Help with an Issue, we see different questions.

Why can't I get AdSense for my blog?

or

Why did Blogger delete my blog?

or maybe

Why can't I find my blog listed, in Google Search?

And all of these questions may well be just different views, of the same problem.

What you can do, and what you should do, with a Blogger blog, may be two different things.

Other Google services may affect success of your Blogger blog.

A Blogger blog may be subject to standards enforced by different Google services.

• AdSense.
• Blogger.
• Google Search / Webmasters.

What you do with your blog, that may be permitted (or tolerated) in one service, may cause problems with another service.

Too many people start a Blogger blog, and only focus on making money. When they don't have a plan for writing content, they try other, more innovative techniques - and there is where many spam blogs start.

AdSense, the Google ad host service, may be less tolerant of innovative techniques, than Blogger. They have conservative standards for blog content, and for behaviour by blog readers. If you include AdSense ads in a Blogger blog, you may find your blog subject to penalties for excessive / self clicking and low quality content, that exceed Blogger standards.

Even without AdSense being involved, a Blogger blog may be subject to search engine penalties for infrequently updated or non unique content. Low quality content, resulting in poor search results, will leave the blog lacking traffic. If the blog uses a third party ad hosting service, which is more tolerant than AdSense about clicking and content, the ads still won't make a lot of money without traffic.

Don't use your Blogger blog as a gateway to a non Google website.

A static blog, that is designed to generate traffic for a non Blogger / Google website, may be a problem, too. Gateway blogs are a popular technique for many spammers. Blogger actively forbids both gateway blogs, and automated posting. This includes feeds from other websites, when used as post content.

You can combine your Blogger blog with a non Google website - but publish the blog as a blog.

It may be possible to have blog posts visible in a section of the blog - and combine the blog, with a non Google website. Even so, the "blog" portion of the blog will still need regularly published, unique content. This is a requirement, whether you are publishing a Blogger blog - or moving your blog to other hosting.

What you can do - and what you should do - will always differ.

The bottom line is what you can do, with a Blogger blog - and what will lead to a successful blog - may be two different things. You're allowed to do anything, that does not violate TOS - but if you want good results, you need to use common sense.

Blogger Blogs Use The Posts Newsfeed, As A Sitemap

Some Blogger blog owners don't know how to setup a sitemap, for their blog.

Setting up the sitemap, for a website, is a major process - and takes time. Every time you add a page to a website, the sitemap has to be updated - or how do your readers find the new page?

Alternatively, you can use a sitemap builder service, which builds the sitemap. This gives you a file, hosted by the sitemap builder service. Will the sitemap builder service be in business next week?

In either case, you take your sitemap file, and upload it to the blog. Every time you add a page, you update the sitemap, then you upload the updated sitemap. Every time - or the new page remains unindexed, until you do.

---

(Update 2014/11/04): Blogger is now providing an automatically generated, dedicated sitemap, to replace the newsfeed sourced sitemap - and an automatic "robots.txt" update. There is no need for a custom sitemap - nor to update "robots.txt".

---

With a Blogger blog, you designate the sitemap, using Google Webmaster Tools, referencing the URL of the posts newsfeed.

You can substitute any custom sitemap, if you wish, but this will be at your own risk - and you can develop your own installation instructions.

Once done with the Webmaster Tools sitemap wizard, you are free to work on the visible parts of the blog. Every time you publish a new post, the post is updated into the newsfeed, and becomes part of the sitemap.

The next time a search engine bot hits the blog, it picks up the new sitemap - and indexes the new post.

Combine the automatically and immediately updated sitemap, with more time spent updating the blog, and you get happier readers, and better search engine reputation. That's a win - win.

If your blog gets very large, you can add a new sitemap segment - at your convenience - for every 500 posts.

Why spend time manually updating a sitemap, with every new post (and worry about a file, hosted by someone who may or may not stay in business)? Use the posts newsfeed (hosted by Google), and work on the blog content.

Using A Robust Sitemap, With Your Blog

For a Blogger blog, proper indexing by the search engines is critical to the success of the blog, in getting readers.

Some blog owners are disappointed to find that their blog has no page rank, and no visibility in the search engine results - and little to no chances for getting readers.

Not all blog owners understand details about the custom domain migration process, any issues related to renaming the blog, or simply how to get a blog properly indexed.

One of the most useful tools, that you as a blog owner can use, is the sitemap.

With a Blogger blog, that is typically a collection of posts indexed by date and maybe by topic, the sitemap, properly presented to the search engines, helps the search engines index all of the posts methodically.

Submit a Sitemap to tell Google about pages on your site we might not otherwise discover.

That's the simple advice, from Google, which you should see.

When I created this blog, and submitted my first sitemap, Google setup a default.

http://bloggerstatusforreal.blogspot.com/feeds/posts/default?orderby=updated

Had you looked at the "robots.txt" file, when this blog was published as "bloggerstatusforreal.blogspot.com", that's what you would have seen.

Having later published this blog as "blogging.nitecruzr.net", Google setup a new default sitemap.

http://blogging.nitecruzr.net/feeds/posts/default?orderby=updated

You may look at my current "robots.txt" file, and that's what you will see.

The default sitemap indexes the most recent 26 posts. For most blogs, especially new ones, you will seldom publish more than that number between each indexing pass by the search engines - and a 26 posts submission is generally sufficient.

For a blog that's been published for a while, and has more than 26 posts, you will need a more robust sitemap than the default. When you change the URL of the blog, or when you use Jump Break on main page posts, you need a sitemap complement that indexes each individual post, one by one.

Note that any sitemap will be much more useful, with the blog publishing a "Full" feed.

In the example of this blog, which now has over 1,500 posts, I would add 4 sitemaps (each sitemap to submit a maximum of 500 posts)

1. http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1&max-results=500
2. http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=501&max-results=500
3. http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1001&max-results=500
4. http://blogging.nitecruzr.net/atom.xml?redirect=false&start-index=1501&max-results=500

Having added the right complement of sitemaps, you watch the display in the Sitemaps wizard:

Submitted URLs
1,539
1,535 URLs in web index

Having submitted those sitemaps last week, 1,535 out of 1,539 posts, allowing for some search engine indexing latency, is about right.

Your blog, having a different number of posts, may use a different complement of sitemaps - but you can use a similar strategy, in determining the number to use. And however you look at it, adding a new sitemap segment for every 500 posts beats manually updating the sitemap for every new post.

>> Top

Search Engine Results Are Not Permanent

One of the oddest problem reports in Blogger Help Forum: Something Is Broken comes from blog owners who only want their blog to be found, in search engine results.

I started a blog 6 months ago. I spent a week getting my blog publicised, it started showing up in search engine results, and all was well. 6 months later, my blog shows up nowhere, it's like it doesn't exist. What happened to my blog??

These blog owners do not realise that a position in any page of the search results is not permanent.

Your blog, to get more readers, needs to be indexed by the search engines - and to appear in search results, which your potential readers use.

Each Search Results Page has only 10 entries.
Each page of search results, for any search query, has only 10 entries. In order for your blog to show up on Page 1 of any search hit list, this month, one entry, that was there last month, now appears on Page 2 of that search hit list.

The owner of the blog recently demoted to Page 2 in the search hit list now has to spend some time getting his blog publicised, so his blog will appear again, on page 1. Next week, a third person is going to find his blog demoted to Page 2.

During the churn over Page 1 and 2, somebody who last month was happy to find her blog listed on Page 2, now finds it listed on Page 3. And people on Pages 3 and 4 are constantly struggling to get their blog to Page 2 and 3, respectively.

Nobody is guaranteed any desired position, in any page for any search.
The lesson here is simple - any position in the search hit list, for any search query, is not permanent - nor will it happen immediately. Similar to the churn over Follower count, when your blog gets a better position, that's because another blog is now getting a worse position.

You will get a good position from hard work, more than from imaginative techniques.

Use Search Console / Webmaster Tools to monitor search activity.
While you are busy checking the search hit lists for the appearance of your blog, you should be checking the diagnostic tools in Google Webmaster Tools. The problem that you see, reflected in the search hit list this week, may have been visible in one or more diagnostic reports last week.

You need to monitor your blog proactively, using Google Webmaster Tools, as much as reactively, using search engine results.

What Is The Mysterious 90 Day Period, After Blog Deletion?

We see periodic evidence of confusion, in Blogger Help Forum: How Do I?, about deletion of our blogs.

I deleted my blog last year - can I get it back?

and

I deleted my blog last month - when will I be able to use it with my new Blogger account?

and

I deleted my blog, but it's still visible, on the Internet!

Not everybody is fully able to understand the "90 day" deletion policy.

Confusion about blog deletion, blog recovery, URL availability, and ongoing visibility have been a problem for many years.

The uncertainty started in 2011, when Blogger gave us the ability to restore our deleted blogs. That year, Blogger gave us the "90 day" provision.

From what we know, the "90 day" provision simply says that blogs deleted over 90 days ago, by the owner, cannot be un deleted. Blogger has yet to advise us, definitively, that a blog (URL), deleted over 90 days ago, would produce any secondary result.

• The URL won't be available, for use by you (under another Blogger account) - or by someone else.
• Any removal from search engine cache will happen only if you plan the deletion, before hitting "Delete blog".
• The blog will simply be un recoverable, by us, since Blogger does not archive deleted blogs for un deletion by us, over 90 days.

Further uncertainties:

• If the URL becomes available, it will be equally available to you, and to everybody else, at the same time.
• If you did not delete the blog, you may not know exactly when the 90 days clock started.
• We are not, with 100% certainty, able to predict that all blogs become available, even after the 90 days.

So don't tell your friends of the date when you will have the blog, re published under the URL - if you are depending upon deletion to make the URL available.

The 90 day period may apply only to blogs deleted by the owner. A blog deleted / locked for suspected spam hosting, by Blogger, may remain in that status indefinitely, until reviewed.

When reviewed as a suspected spam host, a blog will be either removed permanently (for righteous spam deletions), or restored (for spurious spam deletions), at the decision of Blogger Support. In neither case will this, reliably, make the URL available to the public.

Whether the blog is visible on our dashboards (or not), and recoverable by us (or not), unless we make preparation for updating services outside Blogger - for instance, Google Webmaster Tools - a deleted blog will be visible in search engine cache, for a long time.

Getting Traffic To Your Blog Involves Indexing

We continue to see evidence of frustration about getting a blog indexed, in Blogger Help Forum: Something Is Broken.

I can find the blog using the URL - but my visitor log shows nobody is reading the blog!

and

My blog was #1 for my title, in Google, 3 months ago! Last month, it dropped out of sight!! Why does Google let people hack their results???

People who report these problems do not understand that getting traffic to the blog involves more than simply getting the blog indexed, using the Author, Title, or URL of the blog.

Getting your blog indexed, so you get useful traffic from the search engines, requires effort.

1. You have to get the blog indexed.
2. You have to get the blog indexed, in searches which people actually use, when searching for blogs to read.
3. You have to get the blog indexed, with good position, in searches which people actually use, when searching for blogs to read.
4. You have to repeat #3, constantly - because you are competing with everybody else who wants their blog indexed, with good position.

Note that your blog will be indexed faster, if the search engines can read the content. If you have a custom domain, it's best to set the domain up, properly.

How useful is it, to index the blog by title or URL?

Getting the blog indexed, so you can find the blog by searching for the title or URL, gets the blog indexed by the title or the URL. Use Search Console, and look at the "Search queries" list on the dashboard.

How many Impressions do you see, which reference the blog by the Title or the URL? How many Impressions are from people, other than you, checking to see if the blog is indexed?

How valuable will the blog be, if you publish to a vanity domain - which may (or may not) give you a unique URL?

How useful is it, to have your blog indexed, using terms that people search?

If you want new viewers, you need to get the blog indexed, appearing in a search engine results page (aka "SERP") that people (besides you) are using - and appearing in a good position on that page.

To benefit from any search used by a potential viewer, your blog needs to appear at the top of the search list.

• Any position on SERP Page 1 is better than any position on SERP Page 2.
• SERP Page 1 Position 1 is better than Page 1 Position 2.

Which SERP positions, when clicked, yield the most traffic?

All SERP positions are not going to produce the same amount of traffic, to your blog.

Look at the observations, discussing SERP Page One results, from Click Distribution & Percentages by Search Engine Results Page (SERP) Rank. Which positions, on Page One, get more clicks - and more traffic to your blog?

Position #1: 45.46% of all clicks
Position #2: 15.69% of all clicks
Position #3: 10.09% of all clicks
Position #4: 5.49% of all clicks
Position #5: 5.00% of all clicks
Position #6: 3.94% of all clicks
Position #7: 2.51% of all clicks
Position #8: 2.94% of all clicks
Position #9: 1.97% of all clicks
Position #10: 2.71% of all clicks
Total: 95.91% of all clicks occur on SERP Page One

Your blog, linked from SERP Page One Position One, stands an equal chance of getting a new viewer, than appearing in All Other Positions, combined. And only 1 in 20 viewers will even look beyond SERP Page One.

How different will each SERP page be?

Every different search, from people looking for blogs to read, will produce a different list of blogs.

No matter what the subject - or search terms - there can be only one blog, linked from Page One Position One.

• The more potential readers, searching using a given subject or search term, the more readers you have a chance to get.
• The more potential readers, searching using a given subject or search term, the more other people will publish their own blogs to that subject.
• No matter how much hard work you may do with your blog, you are not guaranteed Page 1 Position 1, in any given SERP.

Would you prefer being a small frog, in a large pond - or a large frog, in a small pond? A pond without other frogs - or even with lots of larger frogs - can be a lonely place.

When you have a new blog,you're better off in a community of blogs. As your blog becomes mature, and you have your own reputation, you're better off on your own. You control your domain - and your own destiny.

Try to find a pond which interests you - not one which other people tell you should interest you. And if you're important in your pond, that's good - just don't expect to be important in other ponds, consistently.

How important is unique content?

Finally, if you want new viewers who will return, and who will send you other new viewers, you'll need informative, interesting, and unique content, that is regularly added - and properly targets your potential readers.

Getting Traffc To Your Blog Involves Getting The Blog Properly Indexed

This post has been renamed, as Getting Traffic To Your Blog Involves Getting The Blog Properly Indexed.

Renaming Your Blog, And Using A Sitemap

For a Blogger blog which needs a sitemap for indexing, changing the URL requires creating a new sitemap.

For a blog newly published to a different URL - either a non BlogSpot URL, using custom domain publishing, or simply a different BlogSpot URL - the first few days under the new URL are critical to the success of the blog. Some blog owners find, to their horror, that their blog suddenly has no page rank, and has dropped out of visibility in the search engine results.

I bought a domain from Google, and my blog is now invisible. Obviously, this is another swindle by Google, stealing my money!

These blog owners have not done any research, about the custom domain migration process, or about any issues related to renaming the blog. The successful migration process starts with getting the blog, under its new URL, indexed by the search engines.

One of the most useful tools, that you as a blog owner can use, is a sitemap, now provided automatically, by Blogger.

The new automatic sitemap will index the most recent 2,500 posts. For most blogs, that will be sufficient.

When you re publish your blog under a new URL, you want all posts reindexed promptly. To get all posts reindexed, if your blog contains more than 2,500 posts, you are going to need a more robust sitemap complement.

A sitemap, to support a blog with more than 2,500 posts, will require new entries, in Webmaster Tools.

You want your blog, under the new URL, to be visible under that new URL as soon as possible, so setup a proper sitemap, if necessary - then get to work publishing new posts, and increase search engine activity. And, use Jump Break for all new posts, to focus search engine attention on the individual posts.

Your Blog, Schizophrenia, And The Search Engines

One of my most intriguing subjects (to me, anyway) examines the differences between the "root" alias of a BlogSpot blog, and the "www" alias (ie, "bloggerstatusforreal.blogspot.com" and "www.bloggerstatusforreal.blogspot.com").

The fact that they are separately stored in browser cache will cause mysterious results, when the blog is viewed alternately using the two different aliases.

This is not a terrible problem for your readers, when they simply read the blog. It can cause problems with the search engines, though, as they index your blog based upon inlinks from your readers.

When you publicise your BlogSpot blog, you are hopefully careful to use only one of the aliases, consistently.

If this blog was published only to "bloggerstatusforreal.blogspot.com", that is the URL that I would use. But there are reasons why Blogger provides us with the use of the "www" alias too, and that has very likely spawned a community of bloggers who refer to their blogs using the "www" alias.

If your blog is inlinked by a blogger of the latter community, I will bet that some of them are similarly inlinking to the "www" alias on your blog. So now, you are going to have some readers inlinking to the "root" of your blog, and others inlinking to the "www" alias.

What do the search engines do? They index based upon the inlinks, then they see duplicated content between the two diferent addresses, and they levy the infamous "duplicated content" penalty. Worse yet - because of the asynchronous cache issue, they're going to also index different content occasionally.

Google has provided us with a solution to this "split personality" problem. If you setup Google Webmaster Tools for your blog, click on the "gear" icon, and select "Site Settings". For "Preferred domain", you'll have the choice to have your blog indexed consistently, as either the "root" or the "www" alias.

If you specify your preferred domain as http://www.example.com and we find a link to your site that is formatted as http://example.com, we'll treat that link as if it was http://www.example.com.

You should note that, if you're using a custom domain published blog, as in "www.example.com", this option is more likely to work if you setup the domain DNS properly.

A very simple setting, easy to make, and with apparently magical results. No more schizophrenic indexing. Simple, no?
>> Top

Less Traffic To Our Blogs

One of the more active (and frequently controversial) subjects discussed in this blog is getting more traffic to our blogs, also known as getting the blog indexed by the search engines. Not everybody wants their blogs indexed, though.

How do I make sure my blog is NOT found by search engines?

That question is asked, regularly. And, in truth, there is only one truly effective answer.

Don't publish your blog on the Internet.

This is similar to the question (and answer)

How do I keep my blog private?

Short of absolute effectiveness, the most obvious solution is the two Settings - Basic options. And there are other tweaks which you may find, in Google Webmaster Tools.

And, in the spirit of locking the barn after the horses have been stolen, there is the subject of Removing Your Blog From The Search Engines Caches. Search engine caches, like browser caches, won't clear overnight, if ever.

You'll need to note that the entries in "robots.txt", that result from the two Blogger options "Add your blog to our listings?" and "Let search engines find your blog?", are advisory only - not all search engines observe either setting. And the various search engines accept feeds from each other, in various relationships.

>> Top

Blogs Showing The WMX Verification File, Instead Of Main Page

This week, we have various Blogger blogs reported as displaying a mysterious WMX Verification File, when main page view would be expected.

This is the WMX verification file. If you are having problems with the WMX verification for your blog please visit http://help.blogger.com/bin/static.py?page=start.cs

This symptom was first observed Thursday, October 22, and Blogger Support is aware of the problem. Supposedly, the problem was fixed, though some blog owners are still reporting it.

We note that the Google Webmaster Central Blog: Verifying a Blogger blog in Webmaster Tools now instructs us to use the meta tag verification procedure.

The "Webmaster Tools" auto-verification link from the Blogger dashboard is no longer working and will soon be removed.

Is your blog showing this, in place of main page view? If you can provide some diagnostic details, you might help Blogger to help you and others. Precision here is good - and date, time, and timezone will be appreciated.

• What's the URL of the blog?
• When did you first observe the problem?
• When did you last publish a post to the blog?
• When did you last modify the template?
• Did you add the blog to Google Webmaster Tools, using the dashboard wizard, or using the GWT native procedure?
• When did you add the blog to GWT?

(Update 10/23): Some bloggers are working around the problem by refreshing the blog. From Settings - Publishing,

• Change the name to any available URL, and Save.
• Change the name back to the correct URL, and Save.
• Note for a custom domain published blog, you'll have to publish back to BlogSpot, then back to the correct domain URL.

If you do this, you ought to examine the Google Webmaster Tools entry for the blog, and check your Following relationships carefully. Both GWT and Following are sensitive to blog renames.

>> Top

Verifying Ownership Of Your Site To Google Webmaster Tools

As a web master (blog master), besides adding posts and enhancing the template, you have a third activity which is almost as time consuming.

You will be constantly connecting your web site (blog) to other web sites and blogs. In many cases, you'll simply send the owners of the other web site (blog) a request to link to your blog.

For links to some web sites, you'll not use email, you'll use an online wizard. To prove ownership of your web site, instead of sending email, you'll add a file of specific name, or maybe specific content, into the web site. You can't add a file to your blog, though - with blogs, you have no control over files and folders - all that you do is publish posts.

One of the most important web sites that you can add your blog to is Google Webmaster Tools.

"Webmaster Tools" - now known as "Search Console" - provides you the web site (blog) owner the ability to monitor and tune the relationship between your web site (blog) and the Google Search Engine.

Note that some time ago, Blogger added a quick, universal verification of your blogs. At the bottom of your dashboard, under "Tools and Resources", you'll find a link "Webmaster Tools". This appears to generate a file known as the "WMX Verification File".

Right now, there are some odd circumstances which may be a result of using that link. For right now, I would only verify my blogs, one at a time, adding a meta record - as shown below.

When you verify ownership of your web site to GWT, you add a file of specific name and content, to the root. When you add your blog, you can't do that, so you add a record of specific content - a meta record - to the template. Note that the automated process which reads the blog header, and verifies your ownership, appears to be sensitive to private blog settings.

This meta record, which provides your verification of ownership to Google Webmaster Tools, may be the most important single line of code in the template. Add this with care.

1. Sign in to your Google account.
2. You start from Google Webmaster Tools - Dashboard.
3. In the Click here to add a site box, enter the URL to your blog, and then hit Add Site. Make this entry identical to the URL to which the blog is published. Use the following examples, substituting appropriately your value for "myblog", "mydomain", "xxxxx", and "zzz".
   • If the blog is published to "myblog.blogspot.com", enter it as "myblog.blogspot.com".
   • The blog will never be published to "www.myblog.blogspot.com", so do not enter it as "www.myblog.blogspot.com".
   • If the blog is published to "mydomain.zzz", enter it as "mydomain.zzz".
   • If the blog is published to "www.mydomain.zzz", enter it as "www.mydomain.zzz".
   • If the blog is published to "xxxxx.mydomain.zzz", enter it as "xxxxx.mydomain.zzz".
   If you rename the blog, either changing the BlogSpot URL or republishing to a Custom Domain, remember to repeat this exercise with the updated URL.
4. This will take you to Overview.
   

   Next Step
   Verify your site. By verifying your site you can access comprehensive statistics and crawl errors about the pages in your site

5. Click the link for "Verify your site".
6. This will take you to Verify a Site.
7. In the Choose verification method... pull down list, select Add a meta tag.
8. This will generate your site verification meta tag. Highlight and Copy the text from the box.
9. In a separate window, edit the template for your blog. Add into the template, in the heading section, your site verification meta tag, that's provided by the GWT wizard. After you add the meta tag, you should have something like:
   

   <head>
       <b:include data='blog' name='all-head-content'/>
       <title><data:blog.pageTitle/></title>
       <meta name="description" content="......."></meta>
       <meta name="keywords" content="......."></meta>
       <meta name="verify-v1" content="......."></meta>
   
       <b:skin><![CDATA[/*
   

   
   Note that some detail was removed from the above example, and that the above example shows template code for designer and layout templates.
   
   You might want to observe the different ways to format a meta tag, including differences between classic and layouts formats. The XML, in a layouts template, will require a properly closed tag.
   
   • The tag will probably be presented, by the GWT wizard, as

     <meta name="verify-v1" content=".......">

   • One way to code the tag, in a layouts template, would be

     <meta name="verify-v1" content="......."></meta>

   • A second way to code the tag, in a layouts template, would be

     <meta name="verify-v1" content="......." />

10. Save Template Changes.
11. Go back to Verify a Site, and hit Verify.
12. This will take you to Overview. Part of the display will include:
    

    You've successfully verified (your site)

    and may include
    

    No pages from your site are currently included in Google's index.

    and

    You have not submitted any Sitemaps. Submit a Sitemap to help Google discover pages our crawlers might not otherwise find.

    Select the Sitemaps menu entry.

Having verified ownership of your site, you now can add a Google Sitemap, or use the other features of Google Webmaster Tools.

What's In My "robots.txt" File?

When we setup a Blogger blog, we give up control of files and folders, and let Blogger maintain the structure of the blog.

When we publish a Blogger blog (either to BlogSpot, or to a Google Custom Domain), all that we do is post, and maintain the blog template. Even with the files and folders controlled by Blogger, and normally hidden from view, there are ways to examine the contents of some files.

Those who would be better off not examining file content do so anyway, become confused, and stress themselves needlessly. Every week, we read anxious queries

Help! My blog has been hacked!!

or

My robots.txt file is blocking my blog from being indexed!

You can make two Settings changes that are relevant here, but mostly this file is maintained by Blogger code.

Blogger maintains "robots.txt", in each blog, on our behalf.

Occasionally, Blogger makes changes to our blogs in general, and changes the content of "robots.txt" to support the changes made. Recently, changes to "feedReaderJson" necessitated a change to "robots.txt".

Here's the "robots.txt" file for this blog, "blogging.nitecruzr.net", as of 2016/01.

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://blogging.nitecruzr.net/sitemap.xml

There are 3 main entries, in a standard "robots.txt" file.

Here we see 3 entries. For an explanation of some terms, and for a demonstration illustrating the results, please refer to Google Webmaster Tools - "Analyze robots.txt". You may also be enlightened by reading The Web Robots Pages.

• This allows access to all blog components of the blog ("Disallow: (null)"), to the spider "Mediapartners-Google", which is the spider that crawls pages to determine AdSense content. This entry overrides the following entry, for the specified spider.
  

  User-agent: Mediapartners-Google
  Disallow:

• This disallows indexing from all URLs containing "/search"(ie, label searches) - and it allows all other blog URLs.
  

  User-agent: *
  Disallow: /search
  Allow: /

• This defines the URL of the sitemap.
  

  Sitemap: http://blogging.nitecruzr.net/sitemap.xml

  The sitemap is separate from the blog feed, and automatically provided.

You can use Search Console, for an analysis of your "robots.txt" file.

Here's my analysis of "robots.txt" for this blog, run using the wizard. You'll want to view this in full screen mode, zoomed as highly as possible.

Note the hypothetical effects of indexing 3 hypothetical blog URLs, against both "Googlebot" and "Mediapartners-Google", shown at the very bottom.


I've now examined a dozen or so different similar files, for blogs published to both native BlogSpot and custom domains, and excepting the URL of the sitemap, all files have been identical in content. My conclusion is that this is a normal file, and unless we start seeing a flood of complaints about indexing problems, I see no reason to suspect a problem.

So, the next time someone comes to you moaning

My robots.txt file is blocking my blog from being indexed!

you can assure them

No, your "robots.txt" file is normal.

Then, introduce them to Google Webmaster Tools, and its many diagnostic reports.

Google Webmaster Tools And Web Crawl Logs

One of the most useful selections in Google Webmaster Tools, that your readers will appreciate, is the Web Crawl logs. These logs show various problems that your readers might encounter, as they surf through your blog.

Of these errors, the easiest for you to correct, and possibly the most consistently useful, will be those listed in the "Not found" log.

These are actual coding errors, somewhere in the blog posts. My readers probably find these, from time to time. They shouldn't, though.


So, click on the link "Download this table". That downloads the display, into Excel. Then, copy and paste the first column. That gives me a list.

URL
http://blogging.nitecruzr.net/2...evil-blogs.html
http://blogging.nitecruzr.net/2005/05/troubleshooting-network-neighborhood.html
http://blogging.nitecruzr.net/2006/07/Hyperlink
http://blogging.nitecruzr.net/2006/07/corrupted-templates.html
http://blogging.nitecruzr.net/2006/08/blogger-beta-design-deficiencies.html
http://blogging.nitecruzr.net/2006/08/blogger-beta-problems.html
http://blogging.nitecruzr.net/2006/2007/publicisingyourblog%3E
http://blogging.nitecruzr.net/2007/06/template-corruption-components-missing.html
http://blogging.nitecruzr.net/2007/11/custom-domain-publishing-and-404-error.html.
http://blogging.nitecruzr.net/2008/05/google-custom-domains-two-step-
http://blogging.nitecruzr.net/feedReaderJson

I'm not so sure about the last entry, but the first 10 look resolvable.

>> Top

Google Webmaster Tools And Label Searches

As a publisher of a Blogger blog, besides publishing articles, you have to know how those articles are being read.

Just as knowing who is reading your blog, you need to know who is indexing your blog. The Google Search Engine, which feeds 2/3 of the known major search engines, provides Google Webmaster Tools, so we can monitor how well our blogs are being indexed.

If you're going to use Google Webmaster Tools / Search Console effectively, you need to know how to interpret the reports.

Specifically, you need to know what reports show problems needing attention, and what reports advise us of problems avoided.

Representing the latter, we have a message in the list of URLs restricted by robots.txt, which can be found from a link in "Overview", or from "Diagnostics" - "Web Crawl". Recently, we see email from the Search Console team.

Indexed, though blocked by robots.txt



Note the carefully worded advice.

This means that Index coverage may be negatively affected in Google search results.

With a Blogger blog, "may be" should be "actually, is not".

http://blogging.nitecruzr.net/search/label/Search Engines URL restricted by robots.txt

This list, when run against a blog which has a lot of labels, will be rather long. This will cause panic in the hearts of many. Fortunately, this is unnecessary panic.

The keyword here is

/search/label/

This shows us a label search that is blocked, intentionally by Blogger, so the search engines won't follow it.

Were the search engines to index a label search, they would be indexing the same posts in the blog, that they find from the sitemap. They would then penalise both the sitemap indexed content - and the label search indexed content - in their indexing, for "duplicate content".

With "duplicate content" penalties applied, your posts appear lower in the search results - and your blog gets less new visitors.

So, when you see "... /search/label/ ... URL restricted by robots.txt" in your access report, don't panic. This restriction is to your advantage.

Google Webmaster Tools

As a Google customer, we have access to a very useful set of tools, which help us develop and maintain a healthy relationship between our blogs and the Google search engine spiders.

Google Webmaster Tools (now Search Console) provides us with tools to measure our visibility, and it provides us with helpful hints on how to improve our visibility.

You can use Google Webmaster Tools for any active URL that you administer.

If your blog is published to BlogSpot, you use the "xxxxxxx.blogspot.com" URL; if the blog is published to a custom domain, you use the primary URL for the domain.

Using Google Webmaster Tools for any web site requires that you are the administrator of the web site.

Google doesn't want anybody who's not the administrator, of any web site, accessing search engine diagnostics for that web site.

The meta tag provided in the verification process is a token, and it must match the URL that you are verifying. If you change the active URL for the blog (rename it within "blogspot.com", or publish it to a custom domain), you will need a new Google Webmaster Tools entry, with a new verification tag.

The meta tag is as important a data element as the clickable link that you get in your email, when you're offered membership in a blog - or when you verify an email address, for an account. You click on the link, which opens the token, and applies it to your account.

With a Blogger blog in Google Webmaster Tools, you install the token into the blog template, to verify ownership of the domain (blog). Only a blog administrator has the authority to update the template. If you don't install the token into the template, you can't prove that you own the blog, and you can't use Google Webmaster Tools for your blog.

Elsewhere in Google Webmaster Tools, you'll find the Web crawl logs. Some information in the web crawl logs can be very useful to you, while other is informational only. It's to your advantage to separate the two categories. The "Preferred domain" setting in "Site Configuration" is a very small setting, that can have a big impact. And most of us setup our blogs on GWT so we can add a sitemap to the blog.

>> Top

Custom Feeds And The Google Sitemap

Custom feed services, like FeedBurner, give you much more than a native Blogger blog feed. You can aggregate content and pull in material from other blogs and web sites, you can provide more user friendly formatting, and you can even track the number of readers who subscribe to a FeedBurner custom feed.

Your Blogger blog even supports your use of a custom feed. If you setup the Post Feed Redirect option, all links in the blog that would reference the native blog post feed are replaced by the custom feed. Your blog readers, when they click on a link that says

Click here to subscribe to my feed!

automatically subscribe to the custom feed. This is good for your readers, and for you.

Unfortunately, use of the feed redirect option isn't a good idea for everybody, immediately. If you're using a Google Sitemap, to make it easier for the search engines to index your blog, and bring you more readers, you'll need to use the native blog feed.

With the blog redirecting all blog post feed references, to a custom feed, the sitemap will be broken. The Google Sitemap requires a feed that has the same URL as the blog. A custom feed has the URL of the custom feed service. This breaks the sitemap, unless you modify the feed URL.

Having modified my feed URL, I can create a Google sitemap that the spider can work from.

>> Top

The Blog Feed Redirect Option Needs To Be Used Properly

When you setup a Blogger blog, and use a properly configured feed, the combinations of comments, labels, and posts feeds can do wonders for making your blog more accessible and maintainable. You have a number of feeds, with URLs based upon the URL of the blog, to work with.

For more possibilities, you can use a custom feed, provided by an aggregation service like FeedBurner or FeedDigest. When you use a custom feed, you can even redirect references, within the blog, to the custom feed, so you don't have to manually reconfigure things like post footer feed links.

You redirect feed references using the dashboard setting in Settings - Other - "Post Feed Redirect URL".

Always use this setting properly - only put a valid feed URL in the box, or you will end up with a broken blog feed. A valid feed could be:

• A feed from another Blogger blog, as when you move the blog to another URL, and setup a stub blog with a redirected feed.
• A valid external URL - one provided by FeedBurner, FeedDigest, or a similar service.

Do not point this internally, to the same BlogSpot or domain URL. You cannot use this to simply change the URL of the feed for any one blog - it must be used in conjunction with a second blog, or a custom feed setup.

Here's the redirect setting, for this blog:

http://feeds.feedburner.com/Nitecruzr-Blogging

That's my FeedBurner feed URL. Yours will be similar, excepting the

Nitecruzr-Blogging

If you use the URL of a feed for the same blog, you'll end up with a feed redirect loop. If your readers see anything from the feed, when it's directed to itself, it will be something like

The page isn't redirecting properly

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

If you have no FeedBurner or second blog feed to redirect into, or if you don't know what is needed here, leave this setting blank. If you're using the setting, and you're experiencing a feed related problem, you should first clear the setting.

Also, be aware of the interaction between the Google Sitemap and the blog feed. A Google Sitemap will break, if you redirect your blog feed to a URL outside the blog naming space.

Finally, note that the blog feed is intended to let your readers read your blog, with greater ease. It's not designed to let you randomly distribute the blog content to the Internet, while abusing your Followers and subscribers.