Copy page

Documentation

Multi-Site & Campus

Status: partially implemented (2026-09-06). No second site actually runs against this stack today, so "Site as data, not as infrastructure" and "When mesh-style networking would actually be justified" below are still design reasoning, not built systems. The command channel and MQTT reliability hardening described further down are now implemented — see those sections for what changed. This page still records the reasoning so none of it gets re-litigated (or re-guessed) later.

Site as data, not as infrastructure

Sites own data by topic prefix (see "Who sees which data" below). point_history has no site column, and point identity is a flat urn:point:{topic} namespace (ingest.py).

That flat namespace is actually the reason a campus rollup doesn't need new infrastructure. Point and equip identity are derived straight from the MQTT topic string, and per Windrow's own wire contract the topic is f"{topic_prefix}/{point}" — topic_prefix already varies per Windrow instance (e.g. fbf/mock-ahu-1/...). Point two Windrow instances at two buildings at the same shared mosquitto/mqtt_listener/Postgres/Oxigraph stack, give each a distinct topic_prefix (e.g. fbf/hospital-a/... vs fbf/hospital-b/...), and their points/equipment never collide — no migration, no new column, because identity was never global to begin with.

For campus-level filtering ("all faults across sites," "just Site B"), tag each equip with a site Haystack tag through the ingestion path that already exists (ingest_tags/ingest_equip_tags in ingest.py take an unconstrained tag dict). site is then just a queryable property in Oxigraph like any other tag — no new mechanism.

This only holds if sites can share one Postgres/Oxigraph. If a site must stay data-sovereign or run standalone when disconnected, it needs its own full stack, with the central campus stack pulling from it — the same pull shape haystack_puller/sql_puller already implement, just pointed at a peer Baler instead of a third-party source. That's a bigger decision than this page covers and hasn't come up yet.

Who sees which data

Each site can own one or more topic prefixes (POST /auth/sites with topic_prefixes, or PUT /auth/sites/{id}/topic-prefixes). A site owning fbf/hospital-a owns every point urn:point:fbf/hospital-a/... and every equip urn:equip:fbf/hospital-a/....

• No site anywhere owns a prefix: everyone sees everything. This is a single-tenant install, and it's how every install behaved before scoping existed.
• Your org owns prefixes: you see only those - your token's site_ids' prefixes, or all of your org's if the token carries none.
• Your org owns none (while another does): you see everything except what other orgs claimed. Scoping is per org, so one org partitioning its data doesn't blind the rest.
• Locations (Site, Building, Floor, Zone) have no prefix: one is visible when it contains something you can see (through hasPart, backwards along feeds, hasPoint), and hidden when it contains something another org claimed.
• Out-of-scope records read as missing, never as "forbidden", so their existence isn't revealed. Graph reads apply scope inside the query: to the result and to every record a filter path reaches, so spaceRef->dis == "x" can't test tags on a record you can't see. This applies to the Haystack API (/api: read, nav, hisRead, hisWrite, watches) and to /ingest/his and /ingest/history.
• Prefixes never overlap across orgs. Claiming fbf while another org owns fbf/hospital-a (or the reverse) is rejected, otherwise the broader claim would expose the other org's data.
• First claim wins. In a shared instance, an org can claim a prefix no one owns yet, including one another tenant's devices already publish under. Provision each site's prefixes when you create the site.

Also scoped, by the same rule:

• Faults (/fault/faults): the list shows only faults on your points; acking or snoozing another site's fault is a 404.
• Derivations: a derivation's targets list, dry-run trace and target enable cover only equipment in your sites.
• Validation (/validate/validate): only violations on your points and equipment, and conforms describes those alone.

Shared across the org (configuration, not site data): fault rules, webhooks and derivation definitions.

Known gap: a derivation's output points (urn:point:computed/...) belong to no site prefix, so once scoping is on they don't appear in /api or /ingest/his reads. Read them unscoped (a single-tenant install) until computed points are scoped through the equipment they attach to.

Why not a mesh network

Some building platforms connect sites as a peer-to-peer mesh: every node has its own identity, and any node can route requests (reads, writes, history, watches) to any other. That's the right shape when sites need to talk to each other.

Baler's topology is a star, not a mesh: N sites report up to one gateway/broker/store, and sites never need to talk to each other. A star doesn't need peer identity, routing, or an RPC surface between leaves; a plain broker's client/hub model already covers everything a star topology requires. Building mesh machinery (node keypairs, signed frames, a node-type registry) for a topology that's provably a star would be solving a problem this system doesn't have.

The property that does matter is that the site opens the connection outbound, so it doesn't fight building-network firewalls. MQTT already has that property. Once an MQTT client opens its session to the broker, either side can send over it, and the broker never needs a route back into the site's network. Infrastructure already in this stack covers the one topology this system actually has.

Reaching back into a site: commands, not just ingest

Windrow already has a real REST surface for exactly the kind of thing you'd want to trigger remotely — POST /discover, POST /learn, POST /modbus/discover (takes a CIDR), device/credential management (see Windrow's own docs/bacnet-discovery-api.md and docs/periodic-discovery-and-credentials.md). The gap isn't capability, it's reachability: that server binds inside the building, and a central Baler has no inbound path to it, especially on a network that can only originate connections outward and never receive them.

Implemented (2026-09-06): windrow.api subscribes to a command topic — cmd/{site}/{instance}/# — on the same MQTT connection it already holds open to publish readings, once started with --site-id (see Windrow's own docs/command-channel.md). A command message (e.g. {"action": "modbus_discover", "cidr": "10.2.0.0/24"}) gets dispatched in-process to the same handle_discover/handle_learn/ handle_modbus_discover functions its own HTTP routes call, and the outcome is published to cmd/{site}/{instance}/result. No new server, no new port, no firewall exception — it rides the identical egress rule that already lets Windrow publish telemetry today, because it's the same connection.

The two things this needed to be safe once it controls real scans, not just telemetry, are both done:

• Persistent session (clean_session=False, QoS 1) on Windrow's client, set once --site-id is given (fbf/src/windrow/mqtt_sink.py's connect()) — a command sent while the site is offline is queued by the broker instead of dropped.
• Per-site broker ACLs (mosquitto/acl). allow_anonymous is off; every client authenticates, and a pattern readwrite cmd/%u/# ACL entry means a site's username is its command-topic scope — a client can only read/write its own cmd/<its own username>/#, enforced by the broker itself, not by convention. Live-verified: an authenticated site-a client publishing to cmd/site-b/... is silently dropped by the broker — a subscriber on cmd/site-b/# never receives it.

Edge nodes: per-site broker accounts

A Windrow running at a site (a Raspberry Pi on the building network) connects to this broker from outside the Docker host. It gets its own account, made in Ops → Edge nodes (admin-only; API: POST /auth/broker-accounts, see the auth API's OpenAPI spec):

• One account per node, tied to one site. It may publish only under that site's topic prefixes (<prefix> and <prefix>/…; a prefix is a whole topic level, so fbf/hospital-a doesn't cover fbf/hospital-ab), and publish/subscribe only on its own cmd/<username>/#. It can't read telemetry at all, not even its own site's. The broker enforces this (mosquitto's dynamic-security plugin), so one site's node can't write into another site's points.
• Use the Windrow's --site-id as the username - Windrow's command channel is cmd/{site-id}/{instance}/#, which is the account's cmd/<username>/#. Its --topic-prefix must sit under the site's prefix.
• The password is generated and shown once, with where to connect (BALER_MQTT_PUBLIC_HOST/_PORT/_TLS_PORT in .env). Baler stores only the account's metadata; the broker keeps a salted hash. New password rotates it (the old one stops working at once); Revoke deletes the account and disconnects the node.
• Changing a site's topic prefixes re-applies its nodes' permissions. If the broker is unreachable at that moment the prefixes are still saved and the response says so (broker_error); the next change retries.

Use TLS (port 8883) from outside the host. Set BALER_MQTT_TLS_DIR to a directory holding the certificate (cert.pem + key.pem, or a Caddy-style <host>.crt + <host>.key) and mosquitto also listens on 8883 with TLS, authenticating the same accounts. On nash-srv, setup-runner.sh step 7 points it at the Let's Encrypt cert Caddy already keeps for timberdoodle.nash.engineering and installs a systemd path unit that restarts mosquitto when Caddy renews it; 8883 is published publicly there while plain 1883 stays loopback-only. A node connects with --mqtt-host timberdoodle.nash.engineering --mqtt-port 8883 and TLS on, verifying against the system CA store (it's a public Let's Encrypt cert).

On a Raspberry Pi: Windrow's deploy/pi/install-edge.sh sets a Pi up as a site gateway (Windrow itself), a simulated BACnet or Modbus device, or a simulated weather station, each under systemd with the edge account's credentials in /etc/windrow/edge.env. See Windrow's docs/edge-nodes.md, and Edge nodes for the runbook: which topology to use, connecting the devices, and checking them end to end.

How it fits with the existing accounts: MOSQUITTO_USERS (the baler backend account and any hand-added ones) stay in the password/ACL files. The plugin's own file, dynamic-security.json on the mosquitto-data volume, holds only edge accounts; a client it doesn't know falls through to the files. The baler account manages the plugin over $CONTROL/dynamic-security/v1 (granted in mosquitto/acl), so there's no separate broker admin secret. Back up the mosquitto-data volume with the rest: losing that file loses the edge accounts (re-create them; Ops still lists them, and revoking tolerates an account the broker no longer has).

MQTT delivery guarantees

This matters more once sites are remote and links aren't as reliable as one Docker network. This section used to describe several open gaps; as of 2026-09-06 all of them are closed. QoS 1 publish and disk spooling on Windrow's side, and clean_session=False on mqtt_listener.py, turned out to already be done by the time this page was checked against the code again — the mosquitto queue cap below was the one piece still genuinely open:

• fbf/src/windrow/mqtt_sink.py's publish_reading() publishes at QoS 1, and spools to a local JSONL file (replayed on reconnect) when the client isn't currently connected — a link drop no longer means silently dropped readings on Windrow's side.
• mqtt_listener.py's client uses a stable client_id and clean_session=False, so mosquitto's queued messages actually have a session to be queued against. fault_detector.py and derivation_engine.py authenticate the same way (all three now go through the shared baler.mqtt_util.make_client()) but still use ephemeral sessions — they evaluate live telemetry, not a durable log, so losing a queued backlog across a restart is an accepted gap, unlike the command channel below where a dropped message means a scan silently never runs.
• mosquitto now runs with a real mounted config (mosquitto/mosquitto.conf) setting persistence true and max_queued_messages 200000 (up from the image default of 1000; 10000 until the scale test (#190) showed a listener restart overflowing it at 5k points) — see mosquitto/entrypoint.sh for how the password file is generated from .env at container start rather than committing hashed credentials.
• retain=True is still not a history buffer — it only caches the latest value per topic for new subscribers. The queue depth above is what actually protects an outage window, not retain.

200000 queued messages per client is still not unlimited — a busy multi-point site can still overflow it during a long enough outage. If that turns out to matter in practice, the next step is backfilling from BACnet/Modbus devices' own trend logs on reconnect rather than raising the cap indefinitely.

When mesh-style networking would actually be justified

If Baler deployments themselves needed to federate — separate orgs, data-sovereignty requirements forcing separate stores — and those deployments needed to query or command each other live, not just roll up for viewing, that's when a peer-to-peer mesh starts to earn its cost. That's a stronger requirement than campus aggregation and isn't implied by anything on the roadmap today; if it shows up, revisit this page and design that layer openly (on the Haystack protocol), rather than growing one by accident.