How To Add Custom Robots.txt File in Blogger?

In one of my previous posts, I had discussed Custom Robots Header Tags Settings for blogger. 

If you have read that post then I hope you guys are aware of its importance in search rankings.

Today, I am back with a very useful and must aware blogging term that is Robots.txt.

In blogger it is known as Custom Robots.txt that means now you can customize this file according to your choice.

In today's tutorial, we will discuss this term in deep and come to know about its use and benefits. I will also tell you how to add custom robots.txt file in blogger.

So let's start the guide. 

More...


What is Robots.txt?

Robots.txt is a text file which contains few lines of simple code.

It is saved on the website or blog's server which instruct the web crawlers on how to index and crawl your blog in the search results.

That means you can restrict any web page on your blog from web crawlers so that it can't get indexed in search engines like your blog labels page, your demo page or any other pages that are not as important to get indexed.

Always remember that search crawlers scan the robots.txt file before crawling any web page.

Each blog hosted on blogger has its default robots.txt file which is something look like this: 

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED


Explanation 

This code is divided into three sections. Let's first study each of them after that we will learn how to add custom robots.txt file in blogspot blogs.

User-agent: Mediapartners-Google 

This code is for Google Adsense robots which help them to serve better ads on your blog. Either you are using Google Adsense on your blog or not simply leave it as it is.

User-agent: * 

This is for all robots marked with asterisk (*). In default settings our blog's labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

Disallow: /search 

That means the links having keyword search just after the domain name will be ignored. See below example which is a link of label page named SEO.

http://www.bloggertipstricks.com/search/label/SEO 

And if we remove Disallow: /search from the above code then crawlers will access our entire blog to index and crawl all of its content and web pages.

Here Allow: / refers to the Homepage that means web crawlers can crawl and index our blog's homepage.

Disallow Particular Post

Now suppose if we want to exclude a particular post from indexing then we can add below lines in the code.

Disallow: /yyyy/mm/post-url.html

Here yyyy and mm refers to the publishing year and month of the post respectively. For example if we have published a post in year 2013 in month of March then we have to use below format.

Disallow: /2013/03/post-url.html

To make this task easy, you can simply copy the post URL and remove the blog name from the beginning.

Disallow Particular Page

If we need to disallow a particular page then we can use the same method as above. Simply copy the page URL and remove blog address from it which will something look like this:

Disallow: /p/page-url.html

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED 

This code refers to the sitemap of our blog. By adding sitemap link here we are simply optimizing our blog's crawling rate.

Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present.

Web crawlers will find it easy to crawl all of our posts.

Hence, there are better chances that web crawlers crawl all of our blog posts without ignoring a single one.

Note: This sitemap will only tell the web crawlers about the recent 25 posts. If you want to increase the number of link in your sitemap then replace default sitemap with below one. It will work for first 500 recent posts.

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you have more than 500 published posts in your blog then you can use two sitemaps like below:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000


Adding Custom Robots.Txt to Blogger 

Now the main part of this tutorial is how to add custom robots.txt in blogger. So below are steps to add it.

  1. Go to your blogger blog.
  2. Navigate to Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes
  3. Now paste your robots.txt file code in the box.
  4. Click on Save Changes button.
  5. You are done! 
blogger custom robots txt
  • Save

How to Check Your Robots.txt File? 

You can check this file on your blog by adding /robots.txt at the end of your blog URL in the web browser. For example: 

http://www.yourblogurl.blogspot.com/robots.txt

Once you visit the robots.txt file URL you will see the entire code which you are using in your custom robots.txt file.

See below image.

robots txt
  • Save

Final Words! 

This was the today's complete tutorial on how to add custom robots.txt file in blogger.

I tried my best to make this tutorial as simple and informative as possible. But still if you have any doubt or query then feel free to ask me in the comment section below.

Make sure not to put any code in your custom robots.txt settings without knowing about it. Simply ask me to resolve your queries. I'll tell you everything in detail.

Thanks guys for reading this tutorial. If you liked it then please support me to spread my words by sharing this post on your social media profiles. Happy Blogging!

Categories Blogger
About Ankit Singla

Hi, I'm Ankit Singla Founder of Master Blogging. A blog that helps you master the art of Blogging. Join our Facebook Community and stay connected with other like-minded bloggers.

15 thoughts on “How To Add Custom Robots.txt File in Blogger?”

  1. Nazmul Hasan RAsel

    Thanks a lot. I did it for my blogspot.

  2. Great tutorial.Please what kind of text do I add to the robot.txt to make all my blog post display in search engines or crawlers. Thanks

  3. Hello Ankit Singla Ji,

    iam new for blogging and iam facing some technicle problems like iam net getting trafic from search engines and one more thing recently i went for domain for my blog after that scene is more worst for me…plz help me thank u.

  4. Please Help me

    how to remove blocked by robots txt in blogger

    How to Unblock Robots.txt file ?

    Google not showing my site in search result. And also Index Status
    Blocked by robots

  5. So I just remove Disallow: /search under the User-agent: * code to allow google to index everything in my blog, or do I replace Disallow: /search with Allow: /search ?

  6. Very nice article. Thanks for explaining about robots.txt in a simple manner.

  7. can i remove Disallow: /search in robot.txt please tell me

  8. Hi Ankit, I am blogger and this is the link of my site “http://birthdayhdcard.com/” and I want to add robots.txt to my blog. As you explained how to add robots.txt for blogger but I am using WordPress CMS so how can I add robots.txt to my blog.

  9. HI, what if you have more than 3000 posts? What should I add please?

  10. Hey Ankit I bought custom domain and linked it with blogger about a month ago and then I added basic robots.txt file but when there is some problem. Whenever I search for the robots. txt page and error pops up saying that versatileblogger.com closed unexpectedly. Could you please help and tell where the problem could be?

  11. Nice article sir,
    I would like to ask you,
    How can I create About Me, Contact, and other pages for my website and add it to Navigation Menu.Kindly reply me, sir
    Would you please like to resolve my issue regarding robots.txt, sitemap and robots header tag.

  12. Hello sir, please my website on Google search is showing “A description for this result is not available because of this site’s robots.txt”. It has not been like this since.please I need your help.

  13. I have 40 post on my blog. When I test sitemap with feeds/posts/default?orderby=UPDATED, only 26 submitted status.
    how to resolved that issue.
    thank’s

  14. Hi Ankit, thank you for this tutorial. It was really helpful. I went through the steps and added the code as you have explained in the tutorial but when I tried to check the robots.txt crawler on my blog post I got a message saying “Sorry, the page you were looking for in this blog does not exist.” What should I do?

Leave a Comment

24 Shares