# Post-deploy verification (runbook) After every successful deploy to nash-srv, the Deploy workflow's `verify` job checks the live install by itself: the browser pass, performance budgets, data flow, edge nodes, phone layout and gateway errors during the deploy. It files, updates and closes GitHub issues (label `live-check`), so a person only looks when its job turns red. It is [`tools/live-check`](/live-check) with more modules. It is **report-only and read-only**. It never rolls back or blocks a deploy (the deploy job's status is its own), and the only requests it sends to the install are GETs, the login, and the named edge-node reads (below). Its only writes are GitHub issues, the job summary and the artifact. ## What runs, and when | Step | What | Modules | |---|---|---| | `Verify (cold)` | right after the deploy finishes | `perf` (cold budgets), `browser`, `layout`, `deploy` | | `Let the install warm up` | five minutes, checking every 15 s that no newer deploy has started | | | `Verify (warm)` | five minutes later | `perf` (warm budgets), `dataflow`, `edges` | `perf` runs first in each phase, before the browser warms any cache. Data flow and edge nodes run in the warm phase: a deploy restarts the listener and reconnects nodes, and five minutes later the install should be settled. The job runs in its own queue, so a deploy merged meanwhile doesn't wait for it; if that newer deploy starts, this verification stops as *superseded* (below) rather than measuring a deploy in flight. Each phase writes a table of every check (pass/fail and its numbers) to the job summary, and both phases go into the artifact `live-verify-` (`live-verify-.json`, the summaries and the screenshots), kept 30 days, for comparing deploys. To run it by hand: Actions, Deploy, Run workflow. Tick **verify_only** to check what's live without deploying (there's no deploy window then, so the `deploy` check is skipped). ## The deploy and verify queues The Deploy workflow has two concurrency groups, one per job (#260): | Job | Group | Queue rule | |---|---|---| | `deploy` | `deploy-nash-srv` | one at a time; a burst of merges deploys the running one and then only the newest waiting one (`cancel-in-progress: false`) | | `verify` | `verify-nash-srv` | one at a time (`cancel-in-progress: true`, as a backstop) | So a new deploy waits only for the previous **deploy** job, never for a verification (which used to hold the deploy queue for its whole run, sleep included: 12-18 minutes per deploy). Two deploys never run at once, and neither do two verifications. **Superseded.** A verification checks, before the cold phase, every 15 s during the five-minute wait, and again inside live-check right before it would touch issues, that it is still verifying the newest deploy (`tools/live-check/supersede.mjs`). It is superseded when - the marker the deploy wrapper writes at the end of every deploy, `/var/lib/baler/deployed-commit` (root-owned, world-readable), names another commit than the one this run verifies, or - a newer Deploy run has a `deploy` job that is queued, running or finished successfully (a queued one must still be queued on a second look 15 s later: a run whose deploy is skipped because CI failed shows as queued for a moment). Newer runs whose deploy was skipped, failed or was dropped from the queue don't count. A superseded verification stops cleanly (the job is green, with a step *Superseded by a newer deploy* and a notice saying why). It files, updates and closes **no** issue. A phase it finished stays in the `live-verify-` artifact, marked `phases..run.superseded`, and perf damping skips it: it counts neither for nor against the two-consecutive-breaches rule. The newest deploy of a burst is never superseded, so it always gets the complete cold and warm verification. If the run history can't be read (GitHub API down), the check falls back to the marker alone and says so in a warning, so a hiccup can't suppress the last deploy's verification. A `verify_only` run verifies whatever the marker says is live when it starts. **Runners.** Both jobs run on nash-srv's self-hosted runners, and a runner runs one job at a time. The repo variable `BALER_VERIFY_RUNNER` names the label the verify job asks for: - **Unset (default): one shared runner** (`nash-srv-baler`, label `baler`). A verification still occupies the only runner, but it gives it up as soon as a newer deploy is queued: the wait polls every 15 s (a queued deploy is confirmed by a second look), so the deploy starts within about 30-45 s, except while a cold phase is running (about 2.5 minutes), which finishes first. - **`baler-verify`: a second runner instance** (`nash-srv-baler-verify`, label `baler-verify` and not `baler`, so deploys never land on it). Verification runs beside the deploy runner, so a deploy never waits on it at all, and the supersede check stops a verification as soon as the newer deploy starts. It runs as the same `github-runner` user with its own workspace. That adds no new privilege, but it isn't less privileged either: that user has no docker access, and its existing NOPASSWD sudoers rules (the deploy wrapper, and any others on the host) apply to jobs on this runner too. Treat it with the same trust as the deploy runner. To set up the second runner, once, on nash-srv (needs sudo and a registration token from the repo's Settings, Actions, Runners, New self-hosted runner, valid about an hour): ```bash sudo bash ~github-runner/actions-runner-baler/_work/Baler/Baler/deploy/nash-srv/setup-runner.sh gh variable set BALER_VERIFY_RUNNER --body baler-verify -R jacobnash/Baler # anywhere with gh ``` **Measured on nash-srv (2026-11-08, two runners).** Three merges in about 12 minutes (#262, #263, #264): the second deploy job started 1 s after the first one ended, where it used to wait for the whole verification (7-8 minutes). The verification it overtook stopped in 9 s as superseded, with no issue changes; the last deploy's verification ran complete (cold, wait, warm) with 0 findings; no two deploys overlapped. Results are on #260. The script is idempotent: everything already installed is skipped, and step 8 adds only the verify runner. Set the variable only after the runner shows as online, or verify jobs queue with no runner to take them. To go back to one runner, delete the variable. ## The checks, what a failure means, and what to do Every finding is one issue, titled `[live-check] …`, with the latest numbers in a table, the commit that was deployed, related issues, and the exact command to reproduce it. The fingerprint at the bottom (``) is how the next run finds it again. ### browser (R3) Every UI page at desktop width and the Cx project and run reports, in a real browser: uncaught errors, `console.error`, failed and 4xx/5xx requests, 403s the role shouldn't trigger, the sign-in screen despite a valid token, requests slower than `LIVE_CHECK_SLOW_MS`, and axe-core WCAG 2.1 A/AA violations (serious or critical). See [Live check](/live-check) for the details. **When it fails:** open the issue, then the page it names with the browser's console and network tabs open, signed in as an operator. The screenshots are in the artifact. ### perf (R4): budgets, cold and warm Each route in `tools/live-check/budgets.json` is fetched 5 times (`runs`) through the public gateway with the operator's token. The median and max time and the largest (uncompressed) body are recorded and judged against that phase's budget. | Route | Warm | Cold | Baseline 2026-11-07 (warm / cold) | |---|---|---|---| | `GET /api/read?filter=equip` | median ≤ 0.5 s | ≤ 1.5 s | 0.11-0.29 / 0.71 | | `GET /api/read?filter=point` | ≤ 1.5 s | ≤ 3 s | 0.91-0.95 / 2.99 | | `GET /fault/faults` | ≤ 0.5 s, ≤ 500 kB | ≤ 1 s | 0.08-0.11 / 0.37, 188 kB | | `GET /ops/status` | ≤ 1 s | ≤ 2 s | 0.27-0.38 / 0.83 | | his `readAll(equip)` (`GET /ingest/his?expr=readAll(equip)`) | ≤ 2 s | ≤ 5 s | 1.64-1.65 / 4.92 (#107) | | Cx project report (Home Edge Pilot) | ≤ 2 s | ≤ 5 s | 3.3-3.5 / 10.2, over budget (#150) | A breach is one issue per route and phase (labels `live-check` and `optimization`). It is **damped**: the issue opens (or reopens) only when the route breaches its budget on **two consecutive complete runs of the same phase**. nash-srv's warm timings swing with the box's load (one CPU at a load of 15 has given `GET /api/read?filter=equip` anything from 0.16 to 0.7 s within minutes, #227, #251), and a single slow run shouldn't file an issue. A first breach is listed in the summary as *held* and turns the job red, but changes no issue. An issue that's already open keeps being updated while the route breaches. Closing isn't damped: one complete run of the phase that's back under budget closes it. "The previous run" is the newest earlier `live-verify-` artifact whose phase ran `perf` completely (an incomplete one is skipped, as it would be for closing). The job reads it back through the GitHub API (`gh run download`, the job's `actions: read` permission): the artifact already holds each route's numbers and the run's findings, it outlives the runner, and nothing extra has to be written or kept in sync. The issue table shows that run's numbers as *Previous complete run*. If the artifact can't be read, the summary has an error row `previous run (damping)` (red job) and no perf issue opens until it can; a first run with no earlier artifact holds everything once. `tools/live-check/previous.mjs` has the code, `findings.test.mjs` the tests. **Host load.** Each perf phase records the load average (1, 5 and 15 min, from `/proc/loadavg`) and `nproc` of the machine running the check, before and after timing the routes. In the Deploy workflow that's nash-srv itself (the self-hosted runner), so it's the install's own host. It's in the job summary (`Runner host … during perf`), in the artifact (`phases..perf_host`) and in each perf issue (*Runner host load*). Read a breach against it: a slow number at a load of 15 on one CPU says more about the box than the route. Run by hand from another machine, it's that machine's load. **When it fails:** compare the numbers in the issue with earlier artifacts and with the host load. One slow run right after a deploy is usually a cold cache; a breach in the warm phase is real. Jaeger (`ssh -L 16686:…`, see [nash-srv deploy](/nash-srv-deploy)) shows where the time goes. A route that errors is an error in the summary (the job is red), not an issue, and nothing in that phase closes. **Change a budget:** edit the number in `budgets.json` and merge. The next run judges against it, and if the observed value is now under budget, it closes the issue. Budgets are deliberate: loosen one only when the slower number is accepted, and say why in the commit. ```json { "name": "GET /fault/faults", "path": "/fault/faults", "warm": { "median_s": 0.5, "max_bytes": 500000 }, "cold": { "median_s": 1 }, "refs": [107] } ``` Each route has `path` (or `cx_report`: a Cx project name, whose HTML report is timed; `LIVE_CHECK_PROJECT` overrides it), and per phase any of `median_s`, `max_s` and `max_bytes`. `refs` are related issues, linked from the finding. ### dataflow (R5) Read from `GET /ops/freshness?prefix=…` (ops_api, viewer, scoped to the caller's sites), never from the database: - **Samples are arriving:** rows in `point_history` over the last 60 s are at least `dataflow.rows_last_60s_min` (2900; live runs about 5800 a minute). - **Every point under each prefix is fresh:** each point under `fbf/home/` reported within `stale_factor` (3) times its poll interval, and never needs to be younger than `min_stale_s` (30 s). Baler doesn't store poll intervals (they live in the edge node's connection), so ops_api estimates each one from the point's own samples: the mean spacing over its last hour of data. A point quiet for more than an hour is found through the hourly rollup (up to 7 days back). Stale points are reported per device group, the segment under the prefix: stopping `windrow-modbus` on the modbus Pi gives one finding, `fbf/home/modbus/* points are stale (n of n)`, listing each point's last sample. When it reports again, the next complete run closes it. **When it fails:** check the node in Ops, Connectors (is it online? what does its connection's poll status say?), then the Pi itself (`systemctl status windrow-*`, its MQTT connection). Below the rate floor with every group fresh means points or a whole prefix disappeared: compare `points_seen_last_hour` on Ops with earlier. To watch more prefixes, add `{ "prefix": "fbf/site-b" }` to `dataflow.prefixes`. `expect_groups` lists groups that must exist even if they sent nothing all week (otherwise a group that vanished entirely can't be missed). ### edges (R6) From `GET /ops/edges`, against `tools/live-check/expected-edges.json`: ```json { "max_last_seen_s": 120, "nodes": ["local/local", "home-gateway/modbus"], "announced": { "node": "local/local", "devices": ["bacnet", "modbus"], "check_connections": true } } ``` - every listed node (`site/instance`; `local` is the Windrow beside Baler) is listed, online, and seen within `max_last_seen_s`; - on the `announced` node, each device in `devices` has announced itself and is online (`list_announced`); - every BACnet or Modbus connection on that node with a `device_name` polls the address that device announced (`list_connections`, `modbus_list_connections`). **When it fails:** an offline node is the Pi or its MQTT link (see [Edge nodes](/edge-nodes)). A device that stopped announcing is the device or its `windrow-announce` timer. A connection that doesn't follow its device: check the node's resolver log ("device … moved"). **Add or remove a node:** edit `nodes`. A node that's offline on purpose should come out of the list, not stay red. ### layout (R7) - Every `ui/*.html` in the checkout, as served by the install, has `` (#177). Without it a phone lays the page out 980 px wide and shrinks it, which no overflow check can see. All pages missing it are one finding. - On an iPhone 13 profile (390 px, mobile), no page scrolls sideways, and on the Equipment page the detail pane is at least `layout.min_detail_px` (300) wide after selecting the first point (#160). This pass also reports errors and failed requests that only the phone layout hits. **When it fails:** the issue lists the pages. Open them in the browser's device toolbar at 390 px; the phone screenshots are in the artifact. ### deploy (R8) The gateway answered no 5xx (`deploy.max_5xx`, 0) between the deploy job's start and end. The gateway writes every 5xx it answers, one JSON line each (time, status, method, path; never the query string or headers), to `5xx.log` on the `gateway-logs` volume, and ops_api reads it for `GET /ops/gateway-errors?since=&until=` (operator, whole-org logins only). The finding lists the routes and statuses. **When it fails:** a burst of 502s on one backend means it was recreated before its replacement was healthy (the rolling deploy, #166, should prevent that). A 5xx the gateway didn't answer (Caddy, the gateway itself down) can't be seen here. "the gateway's 5xx log isn't available" means the gateway runs an older `nginx.conf` or lacks the volume: an error, not a pass. ## How it keeps the issue log tidy - A finding with no issue opens one (a perf breach only on its second consecutive complete run, see [perf](#perf-r4-budgets-cold-and-warm)); an open issue is updated in place (*First seen* kept); a closed one is reopened with a comment, **unless it was closed as "not planned"**: a won't-fix is left alone for good. - An issue labelled **`live-check:ignore`** is never touched: not updated, reopened or closed. Use it for a known finding you don't want the job to keep reporting. Ignored and won't-fix findings don't turn the job red. - An open issue that a run no longer sees is closed, but only if the check that owns it (its *scope*: `browser`, `layout`, `perf:cold`, `perf:warm`, `dataflow`, `edges` or `deploy`) ran completely: signed in, and every page, route or endpoint answered. A cold run never closes a warm issue, and an incomplete run closes nothing in that scope. - The job is red when there is any finding that isn't ignored or won't-fix (held perf breaches included) or any check that couldn't run. ## The write guard `tools/live-check/guard.mjs` replaces `fetch` and routes every browser request through one rule: GET, HEAD and OPTIONS pass; so does `POST /auth/login` (from Node only); and so do POSTs to `/ops/edges///actions/` when `` is in `READ_ACTIONS` in `src/baler/edges.py`, read from the checkout (the same list ops_api and the gateway's viewer row use, and a unit test checks they agree). Everything else is refused before it leaves the runner and listed as "blocked" in the summary. The summary and the artifact list every request sent (method and route, ids folded), so a run's log shows exactly what it did. ## One-time setup on nash-srv This needs Jacob (sudo and a credential); the job fails with a message naming the file until it's done. 1. **An operator test account** on the live install, in the demo org: sign in as an admin, Ops, Users, add a user with role **operator** (not admin: it needs no more), for example `live-check@`, with a long random password. 2. **The credentials file**, readable only by the runner user: ```bash sudo -u github-runner mkdir -p ~github-runner/.config/baler sudo -u github-runner install -m 600 /dev/null ~github-runner/.config/baler/live-check.env sudo -u github-runner editor ~github-runner/.config/baler/live-check.env ``` with exactly these three lines (no quotes needed, no spaces around `=`): ``` BALER_TEST_URL=https:// BALER_TEST_EMAIL=live-check@ BALER_TEST_PASSWORD= ``` `BALER_TEST_URL` is the public URL people use (through Caddy), so the timings are what a user sees. Optional lines: `CHROME_PATH=` (a browser binary, if the box has no Google Chrome), `LIVE_CHECK_PROJECT=` (the Cx project to time and open, default `Home Edge Pilot`). 3. **A browser.** The job uses Google Chrome if `google-chrome` is on the runner's PATH; otherwise it downloads Playwright's Chromium into `~github-runner/.cache/ms-playwright` on its first run. If that Chromium can't start (missing system libraries), install them once: `sudo npx -y playwright-core@1.57.0 install-deps chromium`, or install Google Chrome. 4. Nothing else: the job's `GH_TOKEN` is the workflow's own token (`issues: write` and `actions: read`, to read the previous run's artifact, on that job only). The file lives outside the checkout because `actions/checkout` cleans the workspace. It is never printed: the job only checks it exists, is readable, has the three keys, and warns if its mode isn't 600. ## Run it locally ```bash cd tools/live-check npm ci npm test # unit tests and a dry run against a stub install node live-check.mjs --dry-run # every module, report only (perf against warm budgets) node live-check.mjs --dry-run --phase cold --deploy-start 2026-11-07T22:00:00Z --deploy-end 2026-11-07T22:03:00Z node live-check.mjs --dry-run --only perf,edges ``` `--dry-run` changes no issue. Without it, issues are synced with `gh` (signed in, write access), and the previous run of the perf phase is read from the repo's newest `live-verify-*` artifact. | Variable | Default | Meaning | |---|---|---| | `BALER_TEST_ENV` | `~/.config/baler-tests/baler.env` | env file with `BALER_TEST_URL`, `BALER_TEST_EMAIL`, `BALER_TEST_PASSWORD` | | `LIVE_CHECK_BUDGETS` | `budgets.json` | an alternative budgets file (e.g. a local stack's lower ingest floor) | | `LIVE_CHECK_EXPECTED_EDGES` | `expected-edges.json` | an alternative expected-edges file | | `LIVE_CHECK_OUT` | `tools/live-check/out` | where the artifact, summaries and screenshots go | | `LIVE_CHECK_SHA` | `git rev-parse HEAD` | the commit named in the artifact and issues | | `LIVE_CHECK_EXPECT_SHA` | `LIVE_CHECK_SHA` | the commit the deployed-commit marker must name, else the run is superseded (#260) | | `LIVE_CHECK_MARKER` | `/var/lib/baler/deployed-commit` | the deployed-commit marker; a missing file skips that half of the supersede check | | `GITHUB_RUN_ID` + `LIVE_CHECK_REPO` | set in Actions | the run-history half of the supersede check (skipped without them, so a local run is never superseded) | | `LIVE_CHECK_PREVIOUS` | the newest `live-verify-*` artifact | a `live-verify-.json` to use as the previous run for perf damping (also read in a dry run) | | `LIVE_CHECK_DEPLOY_START` / `_END` | none | the deploy window, like `--deploy-start` / `--deploy-end` | | `LIVE_CHECK_SLOW_MS`, `LIVE_CHECK_SETTLE_MS`, `LIVE_CHECK_PROJECT`, `LIVE_CHECK_REPO`, `CHROME_PATH` | | see [Live check](/live-check) | Against a local stack: bring it up with its own project name and volumes (`BALER_VOLUME_PREFIX=verify docker compose -p verify up -d`), create an operator account (`POST /auth/orgs`, then `POST /auth/users` as that admin), point a test env file at `http://127.0.0.1:`, and give it a budgets file with a lower ingest floor. ## Add a check 1. Write `tools/live-check/.mjs` exporting `run(ctx)`. `ctx.api` has `get`, `json`, `time` and `action` (only the edge read actions), `ctx.budgets` and `ctx.expected` are the two JSON files, and `ctx.browser()` gives a Playwright browser. Return `{ scope, complete, checks, raw }`: `checks` are summary rows (`{module, name, status: pass|fail|error|skip, value, budget, detail}`), `raw` are findings (`{kind, key, scope, title, detail, rows?, refs?}`). `kind` and `key` make the fingerprint, so keep numbers out of `key`. 2. Keep the judging pure (`evaluate(...)`) and test it on fixtures in `checks.test.mjs`. 3. Give the `kind` a label in `KIND_LABEL` (`findings.mjs`), register the module in `live-check.mjs` (`MODULES`, `PHASE_MODULES` and the run order), and add it to this page. 4. If it needs data the API doesn't expose, add a small read-only endpoint (like `GET /ops/freshness`) with a policy row in `gateway/njs/policy.js`. Never give the job database or docker access.