If you regularly check your Google Search Console (GSC) account, you might have bumped into a scary-looking warning: "Indexed, despite blocked by robots.txt" or "Blocked by robots.txt" under the Page Indexing report.
Seeing indexing errors can make any blog owner nervous. After all, if Google cannot properly crawl your website, your search rankings and organic traffic can drop significantly. But don't panic—this error is extremely common and very easy to fix once you understand what is happening behind the scenes.
In this comprehensive guide, we will break down what a robots.txt file actually does, why Google flags this error, and how you can fix it step by step on your Blogger or tech website.
What Is a robots.txt File and How Does It Work?
A robots.txt file is a simple text file located at the root folder of your website (for example, yourdomain.com/robots.txt). It serves as an instruction manual for automated web crawlers like Googlebot.
Inside this file, you write specific rules using commands like Allow or Disallow. These rules tell search engine bots which sections of your website they are allowed to visit and which internal paths they should completely stay away from (such as admin pages, search result links, or preview tags).
Why Does the "Blocked by robots.txt" Error Happen?
This error usually pops up due to one of three main reasons:
- Unintentional Disallow Rule: You or a theme template accidentally added a rule like
Disallow: /which blocks your entire website from being crawled. - Blocking Important Sub-pages: Your default Blogger robots.txt file is blocking search result pages (like
/search) or label pages, but Search Console tried to index them anyway. - Outdated Custom Robots File: You pasted a custom code from an old tutorial online that is preventing Google from reading your new post URLs.
Important Note: Blogger automatically manages your default robots.txt file perfectly! In 90% of cases, turning off custom robots.txt and using Google's default code fixes this error instantly.
Step 1: Check Your Current robots.txt File
Before making any changes, take a look at what your current file looks like:
- Open a new browser tab.
- Type your website address followed by
/robots.txt(e.g.,https://www.technicaleks.com/robots.txt). - Press Enter. You will see a few lines of plain text on your screen.
If you see a line that says Disallow: /, it means your entire site is currently blocked from search engines! You need to fix this immediately in your settings.
Step 2: How to Restore Default Blogger robots.txt (Recommended)
Google Blogger already provides an optimized default robots.txt file designed for fast crawling. If you are facing index blocking errors, resetting to default is the safest solution:
- Log in to your Blogger Dashboard.
- Go to Settings from the left menu.
- Scroll down to the Crawlers and indexing section.
- Look for Enable custom robots.txt.
- If the toggle switch is turned ON, click it to turn it OFF.
By turning this off, Blogger will automatically apply its safe default rules, allowing Googlebot to crawl all your main articles and pages without restrictions.
Step 3: Correct Custom robots.txt Code for Blogger (If Required)
If you strictly want to use a Custom robots.txt file for advanced SEO purposes, make sure you use this 100% working code format:
User-agent: * Disallow: /search Allow: / Sitemap: https://www.yourdomain.com/atom.xml?redirect=false&start-index=1&max-results=500
Note: Replace www.yourdomain.com with your actual website URL before saving!
Step 4: Request Re-indexing in Google Search Console
Once you have fixed your robots.txt file, you need to inform Google that the issue has been resolved:
| Action Step | Where to Go | What to Do |
|---|---|---|
| 1. Inspect URL | GSC Top Search Bar | Paste the blocked article link and hit Enter |
| 2. Test Live URL | GSC Right Corner | Click 'Test Live URL' to confirm Google can now access it |
| 3. Request Indexing | GSC Inspection Screen | Click 'Request Indexing' to submit page for re-crawling |
Conclusion
Fixing the "Blocked by robots.txt" error in Google Search Console is quick and straightforward. By keeping your rules clean, removing accidental disallow directives, and submitting the proper sitemap link, you ensure Googlebot can seamlessly index your content and boost your site's search visibility.