Fleet Uptime Monitor — A Green Result Is Only Trustworthy If the List Was Current

On July 21, 2026 roughly 150 client sites returned HTTP 500 and nothing noticed until an unrelated daily job tripped over it. Six days later we found the monitor built to catch that had been passing 18/18 while a stale roster hid 18 live client sites from every probe it had ever run. This agent is what came out of both.

Two incidents, one lesson

July 21: a fleet-wide PHP fatal. The signature that made it hard to see is the important part — the sites returned a fully rendered HTML body along with the 500. Any monitor asking “did a body come back?” would have reported green. The status code is the test.

July 27: the credential cache the monitor read from was found 111 days stale. It had been silently hiding 18 live client sites — including piotrzawislak.com, leopohlmann.com, and matspohlmann.com — from every uptime probe since they launched. The runs were green. The runs were also meaningless.

How it probes

1. Union two rosters, never trust one

A single roster is a single point of blindness. The agent reads both and probes the union:

If one roster is missing, it probes the other and says which one is gone. Never a silent fallback. If both are gone it probes the three anchors alone and says so loudly.

2. Anchors as controls

Three domains are always probed: localservicespotlight.com, blitzmetrics.com, and dennisyu.com. The last two sit on different IPs, which makes them controls — if they are up but a shared-host cohort is down, that is a host-level fault, not the whole internet. Around fifteen more domains are sampled from the union and rotated each run.

3. Real status codes, free

Probing runs as local curl on the Mac, not through a paid SEO API. Uptime checking should never consume research units.

4. Deep-probe to find the layer

Anything not returning 200, or a 301/302 to the same site, gets a four-point deep probe: the front page, /wp-json/, /wp-login.php, and a static asset. Then a DNS lookup, so a shared-host pattern becomes visible.

5. Severity

One site 5xx is a single-site alert. Three or more, or a shared-IP cohort down while a co-hosted control anchor returns 200, is a fleet-wide outage — the July 21 pattern. On that trigger the agent expands the sample to about thirty domains to estimate blast radius.

It drafts. It never sends.

On any failure the agent writes a dated incident file and stages a Gmail draft — never a send. The draft carries the domain-to-status-to-layer table, the one-line diagnosis, the blast radius, and one operational note worth remembering: download zips under /wp-content/uploads/ may still serve while pages are dark. QR codes pointing at a zip keep working; QR codes pointing at a page go dark. Then it verifies the draft exists and reports its id.

The healthy close-out

The usual case is one line, and it has to carry both roster ages so a frozen roster cannot hide behind a green result:

Fleet healthy - 18/18 probed 200/301 (anchors + 15 of N union), checked HH:MM.
Rosters: cache Dd, targets Dd.

Anything over thirty days gets a stale warning appended. Nothing longer, and nothing is drafted — a stale roster is reported here and alerted by the refresh job itself. Drafting an email every three hours would just train everyone to ignore it.

The full skill file

Local filesystem paths are shown relative to the home directory and the alert recipient is redacted; both are absolute and concrete in the working copy.

START

---
name: "fleet-uptime-monitor"
description: "Probe the BlitzAdmin/LSS fleet for HTTP 5xx outages and draft (never send) an alert on a fleet-wide failure. Use for scheduled fleet uptime checks or when asked 'are our sites up / is the fleet down'."
---

# Fleet Uptime Monitor

Catch a fleet-wide outage within the run interval instead of by luck. This exists because on
**2026-07-21 ~150 client sites returned HTTP 500** and nothing noticed until an unrelated daily
job tripped over it. It generalizes the single-site philmershon-uptime-check to the whole fleet.

**Hard rules:** read-only probing; **never send email - drafts only**; healthy = ONE line,
verbose only when something is actually broken; never write a date/status you didn't verify.

## Step 1 - Build the probe set (union TWO rosters - never trust one)
Read both on the LOCAL Mac via Desktop Commander and probe the **union**. A single roster is a
single point of blindness: on 2026-07-27 the credential cache was found 111 days stale, which had
been silently hiding **18 live client sites** from every uptime probe since they launched.

| Roster | Path | Field | Caveat |
|---|---|---|---|
| Credential cache | ~/Documents/Claude/blitzadmin_site_cache.json | sites[] where status in (active, installed) | Written by refresh-site-cache, which needs an interactive BlitzAdmin browser token - **goes stale silently** whenever nobody logs in |
| Publishing targets | ~/Documents/Claude/Projects/Personal Brand Website/Fleet-Publishing-System/targets.json | targets[] | No token required; this is the roster that keeps moving |

- **Report the age of BOTH files in every close-out line - healthy runs included.** A roster that
  stopped updating is itself the incident; it must never be able to rot invisibly again. Flag any
  file >30 days old explicitly.
- Never *silently* fall back. If one roster is missing/unreadable, probe the other and say which
  one is gone. If both are gone, probe the 3 anchors alone and say so loudly.
- Probe set = the 3 ALWAYS-anchors **plus** a random sample of ~15 domains from the union (rotate
  the sample each run):
  - ALWAYS anchors: localservicespotlight.com, blitzmetrics.com, dennisyu.com (blitzmetrics.com +
    dennisyu.com sit on different IPs and act as controls - if they're up but a shared-host cohort
    is down, it's a host-level fault, not the whole internet).

## Step 2 - Probe real HTTP status (free, local curl - NOT DataForSEO units)
Run on the LOCAL Mac via Desktop Commander start_process (never the sandbox). The status CODE is
what matters - the 2026-07-21 fatal returned a fully-rendered HTML body WITH a 500, so "did a body
come back" is the wrong test:

    for u in ; do printf "%s " "$u"; curl -sS -o /dev/null -w "%{http_code}\n" --max-time 20 "https://$u/"; done

Record domain -> front-page status.

## Step 3 - Deep-probe anything non-healthy
Healthy = 200, or 301/302 to the same site. For any domain returning 4xx/5xx (or a timeout),
deep-probe to classify the layer (the philmershon interpretation table):
- / , /wp-json/ , /wp-login.php , and a static asset (a /wp-content/uploads/ file or a theme CSS)
  - record each status.
- dig +short  to get its IP, so you can spot a shared-host pattern.

Interpretation:
- Static asset 200 but / and /wp-json/ are 5xx -> **WordPress/PHP is fataling, web server is fine**
  (the 2026-07-21 signature: a network-activated plugin hooking parse_request/wp/template_redirect).
  wp-login.php and admin-ajax.php may still 200 because they don't run the main query.
- Everything including the static asset fails -> server/network-level (host or DNS), not application.
- Only /wp-json/ or a store path failing -> partial breakage, still urgent.

## Step 4 - Severity
- **All healthy** -> Step 6 (one line).
- **1 site 5xx** -> single-site alert.
- **>=3 sites 5xx, OR a shared-IP cohort down while a co-hosted/control anchor is 200** ->
  **FLEET-WIDE OUTAGE** (the 2026-07-21 pattern). Escalate: expand the sample to ~30 domains to
  estimate blast radius.

## Step 5 - On any failure: draft + record (NEVER send)
1. Write an incident file INCIDENT-fleet-.md in
   ~/Documents/Claude/Projects/Personal Brand Website/ in the style of
   INCIDENT-fleet-500-2026-07-21.md: the down/healthy table, the layer diagnosis, the shared-IP
   finding, likely cause, and the next human step (log into wp-admin and bisect network-activated
   plugins; or read wp-content/debug.log).
2. Stage a Gmail DRAFT to  (connector create_draft, never send): subject
   "Fleet uptime -  sites returning  ()"; body = the domain->status->layer table,
   the one-line diagnosis, blast-radius estimate, and the note that download zips under
   /wp-content/uploads/ may still serve even while pages are dark (QR-to-zip unaffected;
   QR-to-page dark).
3. Verify the draft exists (list drafts) and report its id.

## Step 6 - Healthy close-out (the usual case)
One line only, and it MUST carry both roster ages so a frozen roster can't hide behind a green
result:

    Fleet healthy - 18/18 probed 200/301 (anchors + 15 of  union), checked .
    Rosters: cache d, targets d.

Append "  d stale" for any roster over 30 days. Nothing longer. Do not draft anything -
a stale roster is reported here and alerted by refresh-site-cache itself; drafting every 3h would
just train you to ignore it.

---
Registered as a Cowork skill 2026-07-25 and wired to the fleet-uptime-monitor scheduled task.
Generalizes philmershon-uptime-check; both should call this one rather than re-deriving the probe
logic.

Updated 2026-07-27: probe set now unions the credential cache with targets.json, and every
close-out reports both roster ages. Cause - a healthy 18/18 run was passing while the credential
cache sat 111 days stale, so 18 live client sites had never once been probed. Green results are
only trustworthy if you also know the list was current.