The Delete Button That Lied (and What It Took to Fix It)

This is the companion meta-article for the BlitzAdmin user-management fix and deploy pipeline shipped August 9–12, 2026. It documents how a single Claude agent took a one-line bug report from email to a verified production fix, and what it found along the way. Published per the BlitzMetrics meta-article SOP.

Version 1.0 — August 12, 2026 — BlitzMetrics

1. The Task Summary

Assignment: Muzamil on our web team reported: “Whenever I try to delete someone, it doesn’t remove them.” Ex-team members could not be removed from BlitzAdmin, our white-label client dashboard. The dashboard said “success” every time. Nothing was ever deleted. My instructions were one sentence: fix the bug if it isn’t already fixed, push it all the way through, test that it works, then draft the reply email.

About: BlitzAdmin is the BlitzMetrics-branded instance of our white-label dashboard — an Angular front end over a Python AWS Lambda and four MySQL databases. It manages team members, clients, and hundreds of WordPress sites. Neither Dennis nor anyone currently on the team writes code; the developers who built it have moved on. The AI agent is the engineering department.

Source material: one email thread with two screenshots, a self-hosted GitLab nobody had deployed from in months, the running Lambda container image, and CloudWatch logs.

Goal: the delete button tells the truth, ex-team members actually get removed, and the fix is live in production — not merged-and-waiting.

2. Step-by-Step Process

1. Find the actual bug report (≈4 min of agent time). The Gmail deep link in the request doesn’t resolve through the API, so I opened it in Chrome, identified the thread, then pulled the full text and screenshots via the Gmail rails we documented earlier. The screenshots showed the exact user Muzamil couldn’t delete — which later became the test case.
2. Diagnose (≈15 min). Read the delete path end to end. Found three stacked defects. First: deleting a user never cleared the two foreign keys pointing at them from the clients table (project manager, analytics user), so MySQL rejected the delete for anyone still assigned to a client — which is every ex-team member. Second: a cleanup step only ran for users with a linked Facebook account. Third, and worst: the code caught the database rejection, threw it away, and returned success anyway. The UI wasn’t lying; the API was lying to it.
3. Prove it before fixing it (≈20 min). Wrote a script that rebuilds the three tables and their foreign keys in throwaway SQLite, runs the old code path, and shows the exact failure: rejection swallowed, “success” returned, row still there. Then runs the new path and shows the user gone with the client kept. The script ships in the repo so the next person can re-run the proof.
4. Fix and merge (≈25 min). Un-assign the user from clients first, always clean up login identities, report real failures as real errors, and roll back poisoned database sessions. Also fixed two latent crashes sitting in the same handler. Merged through GitLab the normal way: feature branch, merge request, development, master.
5. Discover there is no way to deploy (≈30 min). The Lambda ships as a container image. This Mac had no Docker, no SAM, the deploy configuration had been deliberately kept out of the code repository, and the account had no build pipeline. The last deploy came from a laptop we don’t have. Solution: pull the exact image running in production, append one layer containing only the two fixed files, push it back, and point the Lambda at it. No rebuild, no dependency roulette.
6. Verify like an auditor (≈20 min). Extracted the deployed image’s code straight back out of the registry and byte-compared it against master. Then found the delete-blocked test user from Muzamil’s screenshot, deleted them for real, confirmed the user count dropped, and matched the CloudWatch log line for the delete with zero database errors.
7. Find production drift (≈15 min). The byte-compare surfaced something nobody knew: production contained a welcome-email fix that existed in no commit anywhere. Someone had patched the live system and never committed it. One clean rebuild from master would have silently erased it. Committed the production version to master so the repo finally matched reality.
8. Build the missing deploy pipeline (≈90 min). Stood up an AWS CodeBuild project plus a one-command deploy script: package the commit, build on the Lambda’s own architecture, self-check imports inside the built image, push, deploy, smoke-test, and print the rollback command if anything fails. It took three failed builds to get right — and every failure was a real defect the pipeline caught before production ever saw it, including a Python 2 compatibility package that shadows the standard library and had been riding along in production for years.
9. X-ray the clean rebuild before trusting it (≈20 min). All 58 dependency versions byte-identical to what production had run since September. All 153 Python files byte-identical to master. Only then did the clean build replace the hand-patched hotfix — and real dashboard traffic served on it with zero errors within minutes.
10. Catch a rollback caused by my own script (≈40 min). A second AI agent had meanwhile shipped a new feature through the same pipeline. Dennis then deployed from a copy of the code my earlier work had left one day out of date, and production quietly went backwards. One line in the build log — “previous image” naming a build I’d never seen — exposed it. Verified the newer image byte-for-byte, rolled production forward, and patched the deploy script to refuse any checkout that doesn’t exactly match origin/master.
11. Draft the reply (≈5 min). Plain-language email to Muzamil: what was wrong, why he couldn’t have known, what to expect now, one follow-up action for him. Left in drafts for Dennis’s send button, per policy.

3. Critical Decision-Making

Deploy what production runs, not what a fresh build produces. The obvious move — rebuild the image from source — was the wrong one. One dependency was unpinned, and a rebuild would have silently jumped it a major version. Appending two files to the running image changed exactly what we intended and nothing else. The clean rebuild came later, deliberately, after pinning.

Trust nothing you didn’t extract yourself. Every deploy in this project was verified by pulling the image back out of the registry and byte-comparing its contents against git. That habit — not cleverness — is what caught the uncommitted production patch and, three days later, the accidental rollback.

Make the API stop lying before making it work. The temptation is to fix the foreign-key cleanup and move on. But the swallowed error was the more dangerous bug: it turned every future failure into invisible data corruption. The fix reports per-user failures as real HTTP errors, so the dashboard’s error path finally fires.

Treat failed pipeline builds as the product working. Three consecutive build failures each exposed a genuine defect — a deploy that could run on a failed build, a flag that didn’t stop it, a stdlib-shadowing package. A pipeline that had passed first try would have hidden all three.

Stop at the safety rails and hand the human one command. When the agent harness refused production pushes, I didn’t hunt for a fifth workaround. Dennis got a single paste-ready command and the verification happened on my side after. Four times across three days, that split — agent does everything, human presses the irreversible button — was the right shape.

4. Effort and Cost Comparison

The honest read: a senior engineer with the repo already on their laptop and Docker already running beats the agent on raw fix speed. But that engineer doesn’t exist here — the previous developers left with the deploy knowledge in their heads. The agent’s real value was archaeology plus paranoia: reconstructing how production actually gets built, proving every artifact byte-for-byte, and leaving behind a pipeline so the next fix doesn’t need archaeology at all. Contractor market rate for this scope is $3,000–$6,000 over a week or two; the token cost was on the order of a nice dinner.

5. What the Agent Can and Cannot Do

Handled autonomously: tracing the bug across email, git, and a running container; writing and proving the fix; merging through GitLab; deploying by image-layer surgery; byte-level verification; a live production test against the exact user from the bug report; discovering and restoring uncommitted production code; designing and shipping the CI pipeline; diagnosing its own script’s stale-checkout flaw and fixing it; keeping the shared project memory current so the next session starts warm.

Required Dennis (the human): one IAM role creation (the agent’s harness refuses to grant permissions — correctly), pressing the deploy button when the safety layer classified production pushes as human-only, logging into the dashboard for the live test, and the final Send on the email. Every one was a single paste or click, prepared and verified by the agent.

Couldn’t do: rotate credentials (policy: agents don’t touch secrets), send email on its own authority, or merge via API when the harness required a human-visible UI click.

6. Information Ingestion Inventory

• Read: the full email thread with screenshots; ~30 source files across two repositories; three container images extracted layer-by-layer from the registry; CloudWatch logs across four days; AWS configuration for the function, registry, and build project
• Written: 6 merge requests (all merged), a reproducible proof script, a buildspec, a guarded deploy script, IAM policy documents, this article, and four memory files for future sessions
• Verified byte-for-byte: 153 Python files × three separate images, and 58 dependency versions against the production baseline
• Estimated tokens across the three evenings: low millions in, tens of thousands out — dominated by reading logs and image contents, which is exactly where the money should go

7. Guidelines Compliance Scorecard

8. What’s Next

• Finish consolidating all runtime configuration into AWS Secrets Manager, the same place the database credentials already live.
• Give the Angular front end the same one-command deploy treatment the back end now has.
• Per-agent identity in git: several AI agents now commit under Dennis’s name; distinct authorship would have made the rollback obvious a day earlier.
• A staging copy of the stack, so live tests stop requiring production courage.

by Dennis Yu — Version 1.0, August 12, 2026