Copy page

Documentation

Post-deploy verification (runbook)

After every successful deploy to nash-srv, the Deploy workflow's verify job checks the live install by itself: the browser pass, performance budgets, data flow, edge nodes, phone layout and gateway errors during the deploy. It files, updates and closes GitHub issues (label live-check), so a person only looks when its job turns red. It is tools/live-check with more modules.

It is report-only and read-only. It never rolls back or blocks a deploy (the deploy job's status is its own), and the only requests it sends to the install are GETs, the login, and the named edge-node reads (below). Its only writes are GitHub issues, the job summary and the artifact.

What runs, and when

Step	What	Modules
Verify (cold)	right after the deploy finishes	perf (cold budgets), browser, layout, deploy
Let the install warm up	five minutes, checking every 15 s that no newer deploy has started
Verify (warm)	five minutes later	perf (warm budgets), dataflow, edges

perf runs first in each phase, before the browser warms any cache. Data flow and edge nodes run in the warm phase: a deploy restarts the listener and reconnects nodes, and five minutes later the install should be settled.

The job runs in its own queue, so a deploy merged meanwhile doesn't wait for it; if that newer deploy starts, this verification stops as superseded (below) rather than measuring a deploy in flight.

Each phase writes a table of every check (pass/fail and its numbers) to the job summary, and both phases go into the artifact live-verify-<sha> (live-verify-<sha>.json, the summaries and the screenshots), kept 30 days, for comparing deploys.

To run it by hand: Actions, Deploy, Run workflow. Tick verify_only to check what's live without deploying (there's no deploy window then, so the deploy check is skipped).

The deploy and verify queues

The Deploy workflow has two concurrency groups, one per job (#260):

Job	Group	Queue rule
deploy	deploy-nash-srv	one at a time; a burst of merges deploys the running one and then only the newest waiting one (cancel-in-progress: false)
verify	verify-nash-srv	one at a time (cancel-in-progress: true, as a backstop)

So a new deploy waits only for the previous deploy job, never for a verification (which used to hold the deploy queue for its whole run, sleep included: 12-18 minutes per deploy). Two deploys never run at once, and neither do two verifications.

Superseded. A verification checks, before the cold phase, every 15 s during the five-minute wait, and again inside live-check right before it would touch issues, that it is still verifying the newest deploy (tools/live-check/supersede.mjs). It is superseded when

• the marker the deploy wrapper writes at the end of every deploy, /var/lib/baler/deployed-commit (root-owned, world-readable), names another commit than the one this run verifies, or
• a newer Deploy run has a deploy job that is queued, running or finished successfully (a queued one must still be queued on a second look 15 s later: a run whose deploy is skipped because CI failed shows as queued for a moment). Newer runs whose deploy was skipped, failed or was dropped from the queue don't count.

A superseded verification stops cleanly (the job is green, with a step Superseded by a newer deploy and a notice saying why). It files, updates and closes no issue. A phase it finished stays in the live-verify-<sha> artifact, marked phases.<phase>.run.superseded, and perf damping skips it: it counts neither for nor against the two-consecutive-breaches rule. The newest deploy of a burst is never superseded, so it always gets the complete cold and warm verification. If the run history can't be read (GitHub API down), the check falls back to the marker alone and says so in a warning, so a hiccup can't suppress the last deploy's verification. A verify_only run verifies whatever the marker says is live when it starts.

Runners. Both jobs run on nash-srv's self-hosted runners, and a runner runs one job at a time. The repo variable BALER_VERIFY_RUNNER names the label the verify job asks for:

• Unset (default): one shared runner (nash-srv-baler, label baler). A verification still occupies the only runner, but it gives it up as soon as a newer deploy is queued: the wait polls every 15 s (a queued deploy is confirmed by a second look), so the deploy starts within about 30-45 s, except while a cold phase is running (about 2.5 minutes), which finishes first.
• baler-verify: a second runner instance (nash-srv-baler-verify, label baler-verify and not baler, so deploys never land on it). Verification runs beside the deploy runner, so a deploy never waits on it at all, and the supersede check stops a verification as soon as the newer deploy starts. It runs as the same github-runner user with its own workspace. That adds no new privilege, but it isn't less privileged either: that user has no docker access, and its existing NOPASSWD sudoers rules (the deploy wrapper, and any others on the host) apply to jobs on this runner too. Treat it with the same trust as the deploy runner.

To set up the second runner, once, on nash-srv (needs sudo and a registration token from the repo's Settings, Actions, Runners, New self-hosted runner, valid about an hour):

Code
sudo bash ~github-runner/actions-runner-baler/_work/Baler/Baler/deploy/nash-srv/setup-runner.sh <RUNNER_REGISTRATION_TOKEN>
gh variable set BALER_VERIFY_RUNNER --body baler-verify -R jacobnash/Baler   # anywhere with gh

Measured on nash-srv (2026-11-08, two runners). Three merges in about 12 minutes (#262, #263, #264): the second deploy job started 1 s after the first one ended, where it used to wait for the whole verification (7-8 minutes). The verification it overtook stopped in 9 s as superseded, with no issue changes; the last deploy's verification ran complete (cold, wait, warm) with 0 findings; no two deploys overlapped. Results are on #260.

The script is idempotent: everything already installed is skipped, and step 8 adds only the verify runner. Set the variable only after the runner shows as online, or verify jobs queue with no runner to take them. To go back to one runner, delete the variable.

The checks, what a failure means, and what to do

Every finding is one issue, titled [live-check] …, with the latest numbers in a table, the commit that was deployed, related issues, and the exact command to reproduce it. The fingerprint at the bottom (<!-- live-check fp:… scope:… -->) is how the next run finds it again.

browser (R3)

Every UI page at desktop width and the Cx project and run reports, in a real browser: uncaught errors, console.error, failed and 4xx/5xx requests, 403s the role shouldn't trigger, the sign-in screen despite a valid token, requests slower than LIVE_CHECK_SLOW_MS, and axe-core WCAG 2.1 A/AA violations (serious or critical). See Live check for the details.

When it fails: open the issue, then the page it names with the browser's console and network tabs open, signed in as an operator. The screenshots are in the artifact.

perf (R4): budgets, cold and warm

Each route in tools/live-check/budgets.json is fetched 5 times (runs) through the public gateway with the operator's token. The median and max time and the largest (uncompressed) body are recorded and judged against that phase's budget.

Route	Warm	Cold	Baseline 2026-11-07 (warm / cold)
GET /api/read?filter=equip	median ≤ 0.5 s	≤ 1.5 s	0.11-0.29 / 0.71
GET /api/read?filter=point	≤ 1.5 s	≤ 3 s	0.91-0.95 / 2.99
GET /fault/faults	≤ 0.5 s, ≤ 500 kB	≤ 1 s	0.08-0.11 / 0.37, 188 kB
GET /ops/status	≤ 1 s	≤ 2 s	0.27-0.38 / 0.83
his readAll(equip) (GET /ingest/his?expr=readAll(equip))	≤ 2 s	≤ 5 s	1.64-1.65 / 4.92 (#107)
Cx project report (Home Edge Pilot)	≤ 2 s	≤ 5 s	3.3-3.5 / 10.2, over budget (#150)

A breach is one issue per route and phase (labels live-check and optimization). It is damped: the issue opens (or reopens) only when the route breaches its budget on two consecutive complete runs of the same phase. nash-srv's warm timings swing with the box's load (one CPU at a load of 15 has given GET /api/read?filter=equip anything from 0.16 to 0.7 s within minutes, #227, #251), and a single slow run shouldn't file an issue. A first breach is listed in the summary as held and turns the job red, but changes no issue. An issue that's already open keeps being updated while the route breaches. Closing isn't damped: one complete run of the phase that's back under budget closes it.

"The previous run" is the newest earlier live-verify-<sha> artifact whose phase ran perf completely (an incomplete one is skipped, as it would be for closing). The job reads it back through the GitHub API (gh run download, the job's actions: read permission): the artifact already holds each route's numbers and the run's findings, it outlives the runner, and nothing extra has to be written or kept in sync. The issue table shows that run's numbers as Previous complete run. If the artifact can't be read, the summary has an error row previous run (damping) (red job) and no perf issue opens until it can; a first run with no earlier artifact holds everything once. tools/live-check/previous.mjs has the code, findings.test.mjs the tests.

Host load. Each perf phase records the load average (1, 5 and 15 min, from /proc/loadavg) and nproc of the machine running the check, before and after timing the routes. In the Deploy workflow that's nash-srv itself (the self-hosted runner), so it's the install's own host. It's in the job summary (Runner host … during perf), in the artifact (phases.<phase>.perf_host) and in each perf issue (Runner host load). Read a breach against it: a slow number at a load of 15 on one CPU says more about the box than the route. Run by hand from another machine, it's that machine's load.

When it fails: compare the numbers in the issue with earlier artifacts and with the host load. One slow run right after a deploy is usually a cold cache; a breach in the warm phase is real. Jaeger (ssh -L 16686:…, see nash-srv deploy) shows where the time goes. A route that errors is an error in the summary (the job is red), not an issue, and nothing in that phase closes.

Change a budget: edit the number in budgets.json and merge. The next run judges against it, and if the observed value is now under budget, it closes the issue. Budgets are deliberate: loosen one only when the slower number is accepted, and say why in the commit.

Code
{ "name": "GET /fault/faults", "path": "/fault/faults",
  "warm": { "median_s": 0.5, "max_bytes": 500000 }, "cold": { "median_s": 1 }, "refs": [107] }

Each route has path (or cx_report: a Cx project name, whose HTML report is timed; LIVE_CHECK_PROJECT overrides it), and per phase any of median_s, max_s and max_bytes. refs are related issues, linked from the finding.

dataflow (R5)

Read from GET /ops/freshness?prefix=… (ops_api, viewer, scoped to the caller's sites), never from the database:

• Samples are arriving: rows in point_history over the last 60 s are at least dataflow.rows_last_60s_min (2900; live runs about 5800 a minute).
• Every point under each prefix is fresh: each point under fbf/home/ reported within stale_factor (3) times its poll interval, and never needs to be younger than min_stale_s (30 s). Baler doesn't store poll intervals (they live in the edge node's connection), so ops_api estimates each one from the point's own samples: the mean spacing over its last hour of data. A point quiet for more than an hour is found through the hourly rollup (up to 7 days back).

Stale points are reported per device group, the segment under the prefix: stopping windrow-modbus on the modbus Pi gives one finding, fbf/home/modbus/* points are stale (n of n), listing each point's last sample. When it reports again, the next complete run closes it.

When it fails: check the node in Ops, Connectors (is it online? what does its connection's poll status say?), then the Pi itself (systemctl status windrow-*, its MQTT connection). Below the rate floor with every group fresh means points or a whole prefix disappeared: compare points_seen_last_hour on Ops with earlier.

To watch more prefixes, add { "prefix": "fbf/site-b" } to dataflow.prefixes. expect_groups lists groups that must exist even if they sent nothing all week (otherwise a group that vanished entirely can't be missed).

edges (R6)

From GET /ops/edges, against tools/live-check/expected-edges.json:

Code
{ "max_last_seen_s": 120,
  "nodes": ["local/local", "home-gateway/modbus"],
  "announced": { "node": "local/local", "devices": ["bacnet", "modbus"], "check_connections": true } }

• every listed node (site/instance; local is the Windrow beside Baler) is listed, online, and seen within max_last_seen_s;
• on the announced node, each device in devices has announced itself and is online (list_announced);
• every BACnet or Modbus connection on that node with a device_name polls the address that device announced (list_connections, modbus_list_connections).

When it fails: an offline node is the Pi or its MQTT link (see Edge nodes). A device that stopped announcing is the device or its windrow-announce timer. A connection that doesn't follow its device: check the node's resolver log ("device … moved").

Add or remove a node: edit nodes. A node that's offline on purpose should come out of the list, not stay red.

layout (R7)

• Every ui/*.html in the checkout, as served by the install, has <meta name="viewport" content="width=device-width, initial-scale=1"> (#177). Without it a phone lays the page out 980 px wide and shrinks it, which no overflow check can see. All pages missing it are one finding.
• On an iPhone 13 profile (390 px, mobile), no page scrolls sideways, and on the Equipment page the detail pane is at least layout.min_detail_px (300) wide after selecting the first point (#160). This pass also reports errors and failed requests that only the phone layout hits.

When it fails: the issue lists the pages. Open them in the browser's device toolbar at 390 px; the phone screenshots are in the artifact.

deploy (R8)

The gateway answered no 5xx (deploy.max_5xx, 0) between the deploy job's start and end. The gateway writes every 5xx it answers, one JSON line each (time, status, method, path; never the query string or headers), to 5xx.log on the gateway-logs volume, and ops_api reads it for GET /ops/gateway-errors?since=&until= (operator, whole-org logins only). The finding lists the routes and statuses.

When it fails: a burst of 502s on one backend means it was recreated before its replacement was healthy (the rolling deploy, #166, should prevent that). A 5xx the gateway didn't answer (Caddy, the gateway itself down) can't be seen here. "the gateway's 5xx log isn't available" means the gateway runs an older nginx.conf or lacks the volume: an error, not a pass.

How it keeps the issue log tidy

• A finding with no issue opens one (a perf breach only on its second consecutive complete run, see perf); an open issue is updated in place (First seen kept); a closed one is reopened with a comment, unless it was closed as "not planned": a won't-fix is left alone for good.
• An issue labelled live-check:ignore is never touched: not updated, reopened or closed. Use it for a known finding you don't want the job to keep reporting. Ignored and won't-fix findings don't turn the job red.
• An open issue that a run no longer sees is closed, but only if the check that owns it (its scope: browser, layout, perf:cold, perf:warm, dataflow, edges or deploy) ran completely: signed in, and every page, route or endpoint answered. A cold run never closes a warm issue, and an incomplete run closes nothing in that scope.
• The job is red when there is any finding that isn't ignored or won't-fix (held perf breaches included) or any check that couldn't run.

The write guard

tools/live-check/guard.mjs replaces fetch and routes every browser request through one rule: GET, HEAD and OPTIONS pass; so does POST /auth/login (from Node only); and so do POSTs to /ops/edges/<site>/<instance>/actions/<action> when <action> is in READ_ACTIONS in src/baler/edges.py, read from the checkout (the same list ops_api and the gateway's viewer row use, and a unit test checks they agree). Everything else is refused before it leaves the runner and listed as "blocked" in the summary. The summary and the artifact list every request sent (method and route, ids folded), so a run's log shows exactly what it did.

One-time setup on nash-srv

This needs Jacob (sudo and a credential); the job fails with a message naming the file until it's done.

1. An operator test account on the live install, in the demo org: sign in as an admin, Ops, Users, add a user with role operator (not admin: it needs no more), for example live-check@<your domain>, with a long random password.

2. The credentials file, readable only by the runner user:

   Code
   sudo -u github-runner mkdir -p ~github-runner/.config/baler
   sudo -u github-runner install -m 600 /dev/null ~github-runner/.config/baler/live-check.env
   sudo -u github-runner editor ~github-runner/.config/baler/live-check.env

   with exactly these three lines (no quotes needed, no spaces around =):

   Code
   BALER_TEST_URL=https://<the live install's public host>
   BALER_TEST_EMAIL=live-check@<your domain>
   BALER_TEST_PASSWORD=<the password>

   BALER_TEST_URL is the public URL people use (through Caddy), so the timings are what a user sees. Optional lines: CHROME_PATH= (a browser binary, if the box has no Google Chrome), LIVE_CHECK_PROJECT= (the Cx project to time and open, default Home Edge Pilot).

3. A browser. The job uses Google Chrome if google-chrome is on the runner's PATH; otherwise it downloads Playwright's Chromium into ~github-runner/.cache/ms-playwright on its first run. If that Chromium can't start (missing system libraries), install them once: sudo npx -y playwright-core@1.57.0 install-deps chromium, or install Google Chrome.

4. Nothing else: the job's GH_TOKEN is the workflow's own token (issues: write and actions: read, to read the previous run's artifact, on that job only).

The file lives outside the checkout because actions/checkout cleans the workspace. It is never printed: the job only checks it exists, is readable, has the three keys, and warns if its mode isn't 600.

Run it locally

Code
cd tools/live-check
npm ci
npm test                                   # unit tests and a dry run against a stub install
node live-check.mjs --dry-run              # every module, report only (perf against warm budgets)
node live-check.mjs --dry-run --phase cold --deploy-start 2026-11-07T22:00:00Z --deploy-end 2026-11-07T22:03:00Z
node live-check.mjs --dry-run --only perf,edges

--dry-run changes no issue. Without it, issues are synced with gh (signed in, write access), and the previous run of the perf phase is read from the repo's newest live-verify-* artifact.

Variable	Default	Meaning
BALER_TEST_ENV	~/.config/baler-tests/baler.env	env file with BALER_TEST_URL, BALER_TEST_EMAIL, BALER_TEST_PASSWORD
LIVE_CHECK_BUDGETS	budgets.json	an alternative budgets file (e.g. a local stack's lower ingest floor)
LIVE_CHECK_EXPECTED_EDGES	expected-edges.json	an alternative expected-edges file
LIVE_CHECK_OUT	tools/live-check/out	where the artifact, summaries and screenshots go
LIVE_CHECK_SHA	git rev-parse HEAD	the commit named in the artifact and issues
LIVE_CHECK_EXPECT_SHA	LIVE_CHECK_SHA	the commit the deployed-commit marker must name, else the run is superseded (#260)
LIVE_CHECK_MARKER	/var/lib/baler/deployed-commit	the deployed-commit marker; a missing file skips that half of the supersede check
GITHUB_RUN_ID + LIVE_CHECK_REPO	set in Actions	the run-history half of the supersede check (skipped without them, so a local run is never superseded)
LIVE_CHECK_PREVIOUS	the newest live-verify-* artifact	a live-verify-<sha>.json to use as the previous run for perf damping (also read in a dry run)
LIVE_CHECK_DEPLOY_START / _END	none	the deploy window, like --deploy-start / --deploy-end
LIVE_CHECK_SLOW_MS, LIVE_CHECK_SETTLE_MS, LIVE_CHECK_PROJECT, LIVE_CHECK_REPO, CHROME_PATH		see Live check

Against a local stack: bring it up with its own project name and volumes (BALER_VOLUME_PREFIX=verify docker compose -p verify up -d), create an operator account (POST /auth/orgs, then POST /auth/users as that admin), point a test env file at http://127.0.0.1:<gateway port>, and give it a budgets file with a lower ingest floor.

Add a check

1. Write tools/live-check/<name>.mjs exporting run(ctx). ctx.api has get, json, time and action (only the edge read actions), ctx.budgets and ctx.expected are the two JSON files, and ctx.browser() gives a Playwright browser. Return { scope, complete, checks, raw }: checks are summary rows ({module, name, status: pass|fail|error|skip, value, budget, detail}), raw are findings ({kind, key, scope, title, detail, rows?, refs?}). kind and key make the fingerprint, so keep numbers out of key.
2. Keep the judging pure (evaluate(...)) and test it on fixtures in checks.test.mjs.
3. Give the kind a label in KIND_LABEL (findings.mjs), register the module in live-check.mjs (MODULES, PHASE_MODULES and the run order), and add it to this page.
4. If it needs data the API doesn't expose, add a small read-only endpoint (like GET /ops/freshness) with a policy row in gateway/njs/policy.js. Never give the job database or docker access.