by /
A single line in BlitzMetrics’s robots.txt file was silently blocking Google from crawling every tag archive page on the site. The fix took under an hour once we knew what to look for — but the issue had likely been suppressing indexing for months.
This type of robots.txt misconfiguration is common on WordPress sites that have both plugin-generated directives and a physical server file. The physical file takes precedence, regardless of the managed provider, so the live /robots.txt response—not the plugin editor—is the source to verify.
💡 Key Takeaway
When a physical robots.txt file exists on the server, WordPress plugins cannot override it. Plugin editors like Rank Math’s built-in robots.txt tool only control a virtual version — which Google ignores when a physical file is present. This is the most commonly missed distinction on managed WordPress hosts.
When a physical robots.txt file exists on the server, WordPress plugins cannot override it. Plugin editors like Rank Math’s built-in robots.txt tool only control a virtual version — which Google ignores when a physical file is present. This is the most commonly missed distinction on managed WordPress hosts.
What We Found
During a Google Search Console audit, we identified over 2,500 pages not being indexed on blitzmetrics.com. One of the root causes was a robots.txt rule that read:
Disallow: */tag/This blocked Google from crawling every tag archive URL on the site. Tag pages pass link equity through to the posts they reference — when Google cannot crawl them, that flow of link signals is cut off.
The rule was not intentional. It had accumulated over time alongside a legitimate rule that we needed to keep:
Disallow: */feed/The */feed/ disallow is correct and should stay. Only the */tag/ line needed to be removed.
Why the Rank Math Editor Couldn’t Fix It
Rank Math SEO has a built-in robots.txt editor. The problem is that WordPress plugins can only control a virtual robots.txt. When a physical robots.txt file exists on the server, it takes priority — and the plugin’s version is completely ignored by Google.
At the time of this May 2026 incident, BlitzMetrics was running on WP Engine. The physical robots.txt file on that server was the authoritative version, which is why editing through Rank Math had no effect.
⚠️ Watch Out
Do not rely on a plugin’s robots.txt editor when a physical robots.txt file exists. Plugin changes will not replace the physical file. Always verify the live response at
Do not rely on a plugin’s robots.txt editor when a physical robots.txt file exists. Plugin changes will not replace the physical file. Always verify the live response at
yourdomain.com/robots.txt.How We Fixed It
The WPCode File Editor (which can directly modify server files) is PRO-only and was not available. Instead, we used WPCode’s PHP snippet manager to write a small snippet using file_put_contents() to overwrite the robots.txt with the corrected content.
The corrected file kept Disallow: */feed/ and removed Disallow: */tag/. After the snippet ran on the next page load, we cleared both the page cache and the CDN cache in the then-current provider portal so Google would receive the updated file rather than a stale cached copy.
We then verified the fix using a cache-busted JavaScript fetch() call with cache: 'no-store' to confirm the live robots.txt reflected the change. Once confirmed, we deactivated and neutralized the WPCode snippet — its job was done.
✅ Pro Tip
After changing robots.txt, purge every active cache layer—including page and edge/CDN caches—and verify the public live file. Exact cache controls vary by platform.
After changing robots.txt, purge every active cache layer—including page and edge/CDN caches—and verify the public live file. Exact cache controls vary by platform.
Critical Decisions Made
Keep the feed disallow: We confirmed the Disallow: */feed/ rule was intentional before touching anything. Removing it could expose RSS feed spam vectors.
Use file_put_contents instead of unlink: Our first approach tried to delete the file entirely. Server permissions blocked deletion silently. Overwriting the file with correct content worked reliably.
Clear each active cache layer: In this incident, clearing only page cache was not enough; the CDN edge cache still served the old robots.txt. We purged both layers in the then-current provider controls and verified the public response.
Effort and Cost Comparison
| Task | Agent Time | Human Time | Agent Cost | Human Cost ($35/hr) |
|---|---|---|---|---|
| GSC diagnosis and root cause identification | ~5 min | 30–60 min | ~$0.08 | $17–$35 |
| robots.txt file investigation | ~2 min | 15–20 min | ~$0.03 | $9–$12 |
| WPCode snippet creation and testing | ~8 min | 45–90 min | ~$0.12 | $26–$53 |
| CDN cache clearing and verification | ~3 min | 10–20 min | ~$0.02 | $6–$12 |
| TOTAL | ~18 min | 1.5–3 hours | ~$0.25 | $58–$112 |
What the Agent Could and Could Not Do
The agent handled the diagnosis, WPCode snippet writing, cache-purge navigation, and verification. Human input was needed for provider-portal access and the decision to authorize changes to a production site.
Guidelines Compliance Scorecard
| BlitzMetrics Guideline | Status |
|---|---|
| Hook opens with specific situation | ✅ PASS |
| Answer in first paragraph | ✅ PASS |
| Written in third person (company site) | ✅ PASS |
| Short paragraphs (3–5 lines max) | ✅ PASS |
| Active voice throughout | ✅ PASS |
| No AI fluff phrases | ✅ PASS |
| Title under 60 chars | ✅ PASS |
| H2/H3 structure without heading abuse | ✅ PASS |
| Internal links to BlitzMetrics content | ✅ PASS |
| Related implementation links added | ✅ PASS |
| Color-coded callout boxes added | ✅ PASS |
| Featured image from real business photo | ⚠️ NEEDS HUMAN |
| RankMath SEO configured | ⚠️ NEEDS HUMAN |
| Categories and tags set | ✅ PASS |
If your site is on WordPress and pages are not being indexed despite good content, check the live robots.txt response and whether a physical file exists—not just the plugin settings. A physical server file overrides the plugin-generated virtual version on any platform.
For the full picture of how this fix fit into BlitzMetrics’s broader indexing audit, see our meta article on setting tag archives to noindex and the full GSC audit meta article.
Dennis Yu
Dennis Yu is the CEO of Local Service Spotlight, a platform that amplifies the reputations of contractors and local service businesses using the Content Factory process. He is a former search engine engineer who has spent a billion dollars on Google and Facebook ads for Nike, Quiznos, Ashley Furniture, Red Bull, State Farm, and other brands.
Dennis has achieved 25% of his goal of creating a million digital marketing jobs by partnering with universities, professional organizations, and agencies. Through Local Service Spotlight, he teaches the Dollar a Day strategy and Content Factory training to help local service businesses enhance their existing local reputation and make the phone ring.
Dennis coaches young adult agency owners serving plumbers, AC technicians, landscapers, roofers, electricians, and believes there should be a standard in measuring local marketing efforts, much like doctors and plumbers must be certified. He has appeared on 353 podcasts with 619 credited episodes — see the full list of his podcast appearances.