Copy page

Documentation

Fault Detection

Fault detection is fault_detector.py: a decoupled third MQTT subscriber plus a periodic sweep of point_history, evaluating rules you configure through the Fault API and recording open/resolved faults in Postgres. It doesn't hook into ingest.py at all — MQTT already fans one message out to every subscriber for free, so a second subscription gets a live copy of every reading with zero coupling to the ingest hot path.

Registering a rule

Code
curl -X POST localhost:8080/fault/rules \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "zone-temp-out-of-range", "mode": "cur", "type": "range", "applies_to": {"topic_glob": "fbf/mock-ahu-1/*"}, "min": 60.0, "max": 80.0}'

applies_to takes exactly one of topic_glob or brick_class. See POST /fault/rules for the full field reference and more stuck/stale examples.

Cur mode only sees MQTT traffic. fault_detector.py subscribes to fbf/# directly and never touches ingest_api.py — so a reading pushed via POST /ingest/ingest (see Architecture) is stored identically, but never evaluated by a Cur-mode rule, since it never crosses MQTT. To trigger one for testing, publish over MQTT yourself, same envelope POST /ingest uses:

Code
mosquitto_pub -h localhost -t 'fbf/mock-ahu-1/analogValue,1' \
  -m '{"point": "fbf/mock-ahu-1/analogValue,1", "value": 95.0}'

Cur vs. His — why there are two evaluation modes

Some faults are only detectable in the moment; others fundamentally need a time window. A range check ("is this value currently outside bounds?") only makes sense against a live reading as it arrives. A stuck or stale check ("has this value stopped changing?", "has this point stopped reporting?") is meaningless without looking back over history — there's no such thing as a single reading being "stale."

So rules declare a mode:

• cur — evaluated in real time, directly off the MQTT message as it arrives. Only range rules run in this mode.
• his — evaluated periodically (every --his-interval seconds, default 30) against point_history. Only stuck/stale rules run in this mode.

Both modes skip samples flagged bad. A reading the device marks as fault, overridden or out of service neither opens nor resolves a Cur-mode fault, and stuck counts only good samples. The same daemon also runs the quality sweep (range, spike, unit), by default every 5 minutes. See Sample quality.

One important asymmetry: a rule's topic_glob is matched against the raw MQTT topic in Cur mode, but against the urn:point:{topic} URI in His mode (since His-mode rules query point_history, which is keyed by URI, not topic). This is fault_detector.py prepending urn:point: onto your topic_glob internally before matching — write topic_glob as the raw topic either way (e.g. "fbf/ahu-1/*", same string a Cur-mode rule would use), never with urn:point: already on the front yourself. Adding the prefix yourself double-prefixes it and the rule silently matches nothing, ever — no error, just zero faults forever. Same glob syntax, different string matched against it under the hood — worth double-checking if a His-mode rule silently never fires.

What each rule type actually checks — and where it doesn't fire

range — faulted if the value is outside [min, max]. Only numeric, non-boolean values can ever match: a None, a string, or a boolean value is treated as not faulted, never as an error. That's deliberate — a sensor going offline and reporting garbage is a different failure mode than a real value being out of bounds, and range isn't the rule that catches it. If you need to know when a point stops reporting sensible values at all, that's what stale is for.

stuck — faulted if every reading in the trailing stuck_seconds window is the same value, and at least one reading exists in that window. A point with zero readings in the window isn't "stuck," it's silent — again, that's stale's job, not stuck's. Pick stuck_seconds longer than the point's natural dwell time at one value, or a legitimately-stable-but-healthy point (an unoccupied zone's flat temperature, say) will false-positive.

stale — faulted if the most recent reading is older than max_age_seconds. A point that has never reported even once is not flagged stale — there's no reliable "first seen" time to measure staleness from, so a point that was misconfigured from day one and never ingested anything will never trigger this rule. It only catches a point that stops reporting after having reported at least once.

The roster is live, not configured

A His-mode rule's set of points to evaluate isn't a separate list you maintain — it's SELECT DISTINCT point_uri FROM point_history filtered by the rule's topic_glob. New points that start reporting are picked up automatically, with zero additional configuration. The flip side is the same as the stale gap above: a point that's never reported never appears in any His-mode roster.

Severity, acknowledging, and snoozing

A rule can carry a free-form severity (e.g. "critical", "warning", "info" — not validated or enumerated, just displayed) alongside name/ mode/type/applies_to. It's copied onto every fault that rule opens, captured at open time — editing a rule's severity afterward doesn't relabel faults it already raised, same as detail.

POST /fault/faults/{id}/ack and POST /fault/faults/{id}/snooze ({"minutes": 60}) let an operator acknowledge or temporarily suppress a specific open fault. Both are pure annotation: acking or snoozing a fault does not stop fault_detector.py from evaluating the underlying rule, close the fault, or affect webhook delivery — a snoozed fault that's still genuinely faulted keeps its status: open and still fires fault.resolved normally if it clears. See POST /faults/{id}/ack for the full field reference.

GET /fault/faults lists faults newest first, one page at a time: 500 by default, up to 5000 with ?limit=. It's still a bare JSON array; the X-Total-Count header says how many match in all, and X-Next-Before (absent on the last page) is the ?before= for the next page. ?since= (ISO 8601 with an offset) narrows to faults that started after a time.

There's a small browser UI over all of this at /ui/alarms.html (through the gateway: localhost:8080/ui/alarms.html) — a live-polling table of faults with Ack/Snooze buttons, filterable by status, plus a "New rule" form and a Rules table with Delete. The form has the same Form/Raw JSON toggle as the derivation editor: Form builds the config field-by-field with a type selector switching between range (min/max), stuck (stuck_seconds), and stale (max_age_seconds) fields; Raw JSON is the same underlying CodeMirror-edited object for hand-writing or pasting a whole rule. It's a thin client over the same API above, nothing it does isn't also reachable by curl.

Controller alarms from BACnet (Windrow forward_alarms)

A Windrow connection with forward_alarms set registers Windrow as a recipient of the controller's own intrinsic-reporting alarms: limits, alarm states and faults the controller itself detects. It publishes them, retained, on {topic_prefix}/{label}/alarm, or on {topic_prefix}/alarms/{type}-{instance} for an object no connection polls. fault_detector turns each one into a fault. No rule is needed.

• One fault per object. Its rule id is bacnet:alarm, and its point is the polled point (or the alarms/... topic). The bacnet: namespace means the sweep that resolves faults of deleted rules leaves these alone.
• Opening and resolving. A transition to any non-normal state (or, after a Windrow restart, an active alarm in Windrow's startup summary) opens the fault. If the fault is already open, its detail and severity are updated instead, for example when it goes from high-limit to low-limit. A transition back to normal resolves it.
• Webhooks. fault.opened and fault.resolved fire as for any rule.
• Severity.
  • A fault state is critical.
  • notify_type event is info.
  • Otherwise, BACnet priority 0-63 is critical, and anything higher is warning.
• Alarms page. The Rule column shows "BACnet alarm (controller)". The Detail column shows the transition, the controller's message text and the object.
• Not readings. These messages aren't stored as readings; mqtt_listener skips them.
• Ack is Baler-only. Acking in Baler doesn't acknowledge the alarm on the controller (Windrow has no AcknowledgeAlarm yet).

Fault lifecycle — transitions only

Opening a fault that's already open is a no-op (no new row, no webhook). Webhooks fire only on the not-open → open and open → resolved transitions — there are no periodic "still faulted" pings while a fault stays open. See Webhooks for delivery semantics.

Upstream: sampling and change-of-value

Fault detection evaluates whatever readings actually arrive — it has no say over how often a point reports or which changes get published in the first place. That's configured upstream, in Windrow's connection layer (his_collect_cov, his_collect_interval, matching real Haystack/Haxall semantics). See Windrow's own bacnet-discovery-api.md (section 3b) for the full writeup, including the one real gap worth knowing before you lean on it: there's no default rate-limit tiering, so an unthrottled noisy point with only his_collect_cov set will publish (and therefore get evaluated) on every single change.