# Fault Detection Fault detection is `fault_detector.py`: a decoupled third MQTT subscriber plus a periodic sweep of `point_history`, evaluating rules you configure through the [Fault API](/api/fault) and recording open/resolved faults in Postgres. It doesn't hook into `ingest.py` at all — MQTT already fans one message out to every subscriber for free, so a second subscription gets a live copy of every reading with zero coupling to the ingest hot path. ## Registering a rule ```bash curl -X POST localhost:8080/fault/rules \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"name": "zone-temp-out-of-range", "mode": "cur", "type": "range", "applies_to": {"topic_glob": "fbf/mock-ahu-1/*"}, "min": 60.0, "max": 80.0}' ``` `applies_to` takes exactly one of `topic_glob` or `brick_class`. See [`POST /fault/rules`](/api/fault) for the full field reference and more `stuck`/`stale` examples. **Cur mode only sees MQTT traffic.** `fault_detector.py` subscribes to `fbf/#` directly and never touches `ingest_api.py` — so a reading pushed via `POST /ingest/ingest` (see [Architecture](/architecture)) is stored identically, but never evaluated by a Cur-mode rule, since it never crosses MQTT. To trigger one for testing, publish over MQTT yourself, same envelope `POST /ingest` uses: ```bash mosquitto_pub -h localhost -t 'fbf/mock-ahu-1/analogValue,1' \ -m '{"point": "fbf/mock-ahu-1/analogValue,1", "value": 95.0}' ``` ## Cur vs. His — why there are two evaluation modes Some faults are only detectable in the moment; others fundamentally need a time window. A `range` check ("is this value currently outside bounds?") only makes sense against a live reading as it arrives. A `stuck` or `stale` check ("has this value stopped changing?", "has this point stopped reporting?") is meaningless without looking back over history — there's no such thing as a single reading being "stale." So rules declare a `mode`: - **`cur`** — evaluated in real time, directly off the MQTT message as it arrives. Only `range` rules run in this mode. - **`his`** — evaluated periodically (every `--his-interval` seconds, default 30) against `point_history`. Only `stuck`/`stale` rules run in this mode. Both modes skip samples flagged bad. A reading the device marks as fault, overridden or out of service neither opens nor resolves a Cur-mode fault, and `stuck` counts only good samples. The same daemon also runs the quality sweep (range, spike, unit), by default every 5 minutes. See [Sample quality](/quality). One important asymmetry: a rule's `topic_glob` is matched against the **raw MQTT topic** in Cur mode, but against the **`urn:point:{topic}` URI** in His mode (since His-mode rules query `point_history`, which is keyed by URI, not topic). This is `fault_detector.py` prepending `urn:point:` onto your `topic_glob` internally before matching — **write `topic_glob` as the raw topic either way** (e.g. `"fbf/ahu-1/*"`, same string a Cur-mode rule would use), never with `urn:point:` already on the front yourself. Adding the prefix yourself double-prefixes it and the rule silently matches nothing, ever — no error, just zero faults forever. Same glob syntax, different string matched against it under the hood — worth double-checking if a His-mode rule silently never fires. ## What each rule type actually checks — and where it doesn't fire **`range`** — faulted if the value is outside `[min, max]`. Only numeric, non-boolean values can ever match: a `None`, a string, or a boolean value is treated as *not faulted*, never as an error. That's deliberate — a sensor going offline and reporting garbage is a different failure mode than a real value being out of bounds, and `range` isn't the rule that catches it. If you need to know when a point stops reporting sensible values at all, that's what `stale` is for. **`stuck`** — faulted if every reading in the trailing `stuck_seconds` window is the same value, *and* at least one reading exists in that window. A point with **zero** readings in the window isn't "stuck," it's silent — again, that's `stale`'s job, not `stuck`'s. Pick `stuck_seconds` longer than the point's natural dwell time at one value, or a legitimately-stable-but-healthy point (an unoccupied zone's flat temperature, say) will false-positive. **`stale`** — faulted if the most recent reading is older than `max_age_seconds`. A point that has **never** reported even once is *not* flagged stale — there's no reliable "first seen" time to measure staleness from, so a point that was misconfigured from day one and never ingested anything will never trigger this rule. It only catches a point that *stops* reporting after having reported at least once. ## The roster is live, not configured A His-mode rule's set of points to evaluate isn't a separate list you maintain — it's `SELECT DISTINCT point_uri FROM point_history` filtered by the rule's `topic_glob`. New points that start reporting are picked up automatically, with zero additional configuration. The flip side is the same as the `stale` gap above: a point that's never reported never appears in any His-mode roster. ## Severity, acknowledging, and snoozing A rule can carry a free-form `severity` (e.g. `"critical"`, `"warning"`, `"info"` — not validated or enumerated, just displayed) alongside `name`/ `mode`/`type`/`applies_to`. It's copied onto every fault that rule opens, captured at open time — editing a rule's severity afterward doesn't relabel faults it already raised, same as `detail`. `POST /fault/faults/{id}/ack` and `POST /fault/faults/{id}/snooze` (`{"minutes": 60}`) let an operator acknowledge or temporarily suppress a specific open fault. Both are pure annotation: acking or snoozing a fault does not stop `fault_detector.py` from evaluating the underlying rule, close the fault, or affect webhook delivery — a snoozed fault that's still genuinely faulted keeps its `status: open` and still fires `fault.resolved` normally if it clears. See [`POST /faults/{id}/ack`](/api/fault) for the full field reference. `GET /fault/faults` lists faults newest first, one page at a time: 500 by default, up to 5000 with `?limit=`. It's still a bare JSON array; the `X-Total-Count` header says how many match in all, and `X-Next-Before` (absent on the last page) is the `?before=` for the next page. `?since=` (ISO 8601 with an offset) narrows to faults that started after a time. There's a small browser UI over all of this at `/ui/alarms.html` (through the gateway: `localhost:8080/ui/alarms.html`) — a live-polling table of faults with Ack/Snooze buttons, filterable by status, plus a "New rule" form and a Rules table with Delete. The form has the same Form/Raw JSON toggle as [the derivation editor](/derivation): Form builds the config field-by-field with a `type` selector switching between `range` (`min`/`max`), `stuck` (`stuck_seconds`), and `stale` (`max_age_seconds`) fields; Raw JSON is the same underlying CodeMirror-edited object for hand-writing or pasting a whole rule. It's a thin client over the same API above, nothing it does isn't also reachable by curl. ## Controller alarms from BACnet (Windrow `forward_alarms`) A Windrow connection with `forward_alarms` set registers Windrow as a recipient of the controller's own intrinsic-reporting alarms: limits, alarm states and faults the controller itself detects. It publishes them, retained, on `{topic_prefix}/{label}/alarm`, or on `{topic_prefix}/alarms/{type}-{instance}` for an object no connection polls. `fault_detector` turns each one into a fault. No rule is needed. - **One fault per object.** Its rule id is `bacnet:alarm`, and its point is the polled point (or the `alarms/...` topic). The `bacnet:` namespace means the sweep that resolves faults of deleted rules leaves these alone. - **Opening and resolving.** A transition to any non-normal state (or, after a Windrow restart, an active alarm in Windrow's startup summary) opens the fault. If the fault is already open, its detail and severity are updated instead, for example when it goes from high-limit to low-limit. A transition back to `normal` resolves it. - **Webhooks.** `fault.opened` and `fault.resolved` fire as for any rule. - **Severity.** - A `fault` state is `critical`. - `notify_type` `event` is `info`. - Otherwise, BACnet priority 0-63 is `critical`, and anything higher is `warning`. - **Alarms page.** The Rule column shows "BACnet alarm (controller)". The Detail column shows the transition, the controller's message text and the object. - **Not readings.** These messages aren't stored as readings; `mqtt_listener` skips them. - **Ack is Baler-only.** Acking in Baler doesn't acknowledge the alarm on the controller (Windrow has no AcknowledgeAlarm yet). ## Fault lifecycle — transitions only Opening a fault that's already open is a no-op (no new row, no webhook). Webhooks fire only on the **not-open → open** and **open → resolved** transitions — there are no periodic "still faulted" pings while a fault stays open. See [Webhooks](/webhooks) for delivery semantics. ## Upstream: sampling and change-of-value Fault detection evaluates whatever readings actually arrive — it has no say over *how often* a point reports or *which* changes get published in the first place. That's configured upstream, in Windrow's connection layer (`his_collect_cov`, `his_collect_interval`, matching real Haystack/Haxall semantics). See Windrow's own `bacnet-discovery-api.md` (section 3b) for the full writeup, including the one real gap worth knowing before you lean on it: there's no default rate-limit tiering, so an unthrottled noisy point with only `his_collect_cov` set will publish (and therefore get evaluated) on every single change.