A single line in BlitzMetrics’s robots.txt file was silently blocking Google from crawling every tag archive page on the site. The fix took under an hour once we knew what to look for — but the issue had likely been suppressing indexing for months.
This type of robots.txt misconfiguration is one of the most common issues BlitzMetrics encounters on client WordPress sites — especially those hosted on managed platforms like WP Engine. The same pattern appeared when we set up RankMath Pro for a local service business on WP Engine — physical file settings consistently override plugin-level settings.
💡 Key Takeaway
When a physical robots.txt file exists on the server, WordPress plugins cannot override it. Plugin editors like Rank Math’s built-in robots.txt tool only control a virtual version — which Google ignores when a physical file is present. This is the most commonly missed distinction on managed WordPress hosts.
When a physical robots.txt file exists on the server, WordPress plugins cannot override it. Plugin editors like Rank Math’s built-in robots.txt tool only control a virtual version — which Google ignores when a physical file is present. This is the most commonly missed distinction on managed WordPress hosts.
What We Found
During a Google Search Console audit, we identified over 2,500 pages not being indexed on blitzmetrics.com. One of the root causes was a robots.txt rule that read:
Disallow: */tag/This blocked Google from crawling every tag archive URL on the site. Tag pages pass link equity through to the posts they reference — when Google cannot crawl them, that flow of link signals is cut off.
The rule was not intentional. It had accumulated over time alongside a legitimate rule that we needed to keep:
Disallow: */feed/The */feed/ disallow is correct and should stay. Only the */tag/ line needed to be removed.
Why the Rank Math Editor Couldn’t Fix It
Rank Math SEO has a built-in robots.txt editor. The problem is that WordPress plugins can only control a virtual robots.txt. When a physical robots.txt file exists on the server, it takes priority — and the plugin’s version is completely ignored by Google.
BlitzMetrics is hosted on WP Engine. The physical robots.txt file on the server was the authoritative version, which is why editing through Rank Math had no effect.
⚠️ Watch Out
Do not rely on Rank Math’s robots.txt editor on WP Engine or any managed WordPress host that has a physical robots.txt file. Your changes in the plugin will not reach Google. Always verify the live file at yourdomain.com/robots.txt directly.
Do not rely on Rank Math’s robots.txt editor on WP Engine or any managed WordPress host that has a physical robots.txt file. Your changes in the plugin will not reach Google. Always verify the live file at yourdomain.com/robots.txt directly.
How We Fixed It
The WPCode File Editor (which can directly modify server files) is PRO-only and was not available. Instead, we used WPCode’s PHP snippet manager to write a small snippet using file_put_contents() to overwrite the robots.txt with the corrected content.
The corrected file kept Disallow: */feed/ and removed Disallow: */tag/. After the snippet ran on the next page load, we cleared both the page cache and the CDN (Network) cache in the WP Engine portal to make sure Google would see the updated version and not a stale cached copy.
We then verified the fix using a cache-busted JavaScript fetch() call with cache: 'no-store' to confirm the live robots.txt reflected the change. Once confirmed, we deactivated and neutralized the WPCode snippet — its job was done.
✅ Pro Tip
Always clear both page cache AND CDN (Network) cache on WP Engine after changing robots.txt. Clearing only page cache leaves the CDN edge serving the old file to Googlebot. Both layers must be flushed for the change to be visible site-wide.
Always clear both page cache AND CDN (Network) cache on WP Engine after changing robots.txt. Clearing only page cache leaves the CDN edge serving the old file to Googlebot. Both layers must be flushed for the change to be visible site-wide.
Critical Decisions Made
Keep the feed disallow: We confirmed the Disallow: */feed/ rule was intentional before touching anything. Removing it could expose RSS feed spam vectors.
Use file_put_contents instead of unlink: Our first approach tried to delete the file entirely. Server permissions blocked deletion silently. Overwriting the file with correct content worked reliably.
Clear CDN separately from page cache: WP Engine has two distinct cache layers. Clearing only page cache was not enough — the CDN edge cache was still serving the old robots.txt. We cleared both through the WP Engine portal’s Network cache option.
Effort and Cost Comparison
| Task | Agent Time | Human Time | Agent Cost | Human Cost ($35/hr) |
|---|---|---|---|---|
| GSC diagnosis and root cause identification | ~5 min | 30–60 min | ~$0.08 | $17–$35 |
| robots.txt file investigation | ~2 min | 15–20 min | ~$0.03 | $9–$12 |
| WPCode snippet creation and testing | ~8 min | 45–90 min | ~$0.12 | $26–$53 |
| CDN cache clearing and verification | ~3 min | 10–20 min | ~$0.02 | $6–$12 |
| TOTAL | ~18 min | 1.5–3 hours | ~$0.25 | $58–$112 |
What the Agent Could and Could Not Do
The agent handled the full diagnosis, WPCode snippet writing, cache purge navigation, and verification independently. Human input was needed only for WP Engine portal login credentials and the initial decision to authorize changes to a production site.
Guidelines Compliance Scorecard
| BlitzMetrics Guideline | Status |
|---|---|
| Hook opens with specific situation | ✅ PASS |
| Answer in first paragraph | ✅ PASS |
| Written in third person (company site) | ✅ PASS |
| Short paragraphs (3–5 lines max) | ✅ PASS |
| Active voice throughout | ✅ PASS |
| No AI fluff phrases | ✅ PASS |
| Title under 60 chars | ✅ PASS |
| H2/H3 structure without heading abuse | ✅ PASS |
| Internal links to BlitzMetrics content | ✅ PASS |
| Client links added (RankMath Pro / WP Engine) | ✅ PASS |
| Color-coded callout boxes added | ✅ PASS |
| Featured image from real business photo | ⚠️ NEEDS HUMAN |
| RankMath SEO configured | ⚠️ NEEDS HUMAN |
| Categories and tags set | ✅ PASS |
If your site is on WordPress and your pages are not getting indexed despite good content, check your physical robots.txt file first — not just your plugin settings. The plugin only controls a virtual version that gets overridden by any file that already exists on the server. This is one of the most commonly missed technical SEO issues on WordPress sites hosted on managed platforms like WP Engine.
For the full picture of how this fix fit into BlitzMetrics’s broader indexing audit, see our meta article on setting tag archives to noindex and the full GSC audit meta article.
Dennis Yu
Dennis Yu is the CEO of Local Service Spotlight, a platform that amplifies the reputations of contractors and local service businesses using the Content Factory process. He is a former search engine engineer who has spent a billion dollars on Google and Facebook ads for Nike, Quiznos, Ashley Furniture, Red Bull, State Farm, and other brands.
Dennis has achieved 25% of his goal of creating a million digital marketing jobs by partnering with universities, professional organizations, and agencies. Through Local Service Spotlight, he teaches the Dollar a Day strategy and Content Factory training to help local service businesses enhance their existing local reputation and make the phone ring.
Dennis coaches young adult agency owners serving plumbers, AC technicians, landscapers, roofers, electricians, and believes there should be a standard in measuring local marketing efforts, much like doctors and plumbers must be certified.