# Multi-Site & Campus **Status: partially implemented (2026-09-06).** No second site actually runs against this stack today, so "Site as data, not as infrastructure" and "When mesh-style networking would actually be justified" below are still design reasoning, not built systems. The command channel and MQTT reliability hardening described further down **are** now implemented — see those sections for what changed. This page still records the reasoning so none of it gets re-litigated (or re-guessed) later. ## Site as data, not as infrastructure Sites own data by **topic prefix** (see "Who sees which data" below). `point_history` has no site column, and point identity is a flat `urn:point:{topic}` namespace (`ingest.py`). That flat namespace is actually the reason a campus rollup doesn't need new infrastructure. Point and equip identity are derived straight from the MQTT topic string, and per [Windrow](/windrow)'s own wire contract the topic is `f"{topic_prefix}/{point}"` — `topic_prefix` already varies per Windrow instance (e.g. `fbf/mock-ahu-1/...`). Point two Windrow instances at two buildings at the same shared `mosquitto`/`mqtt_listener`/Postgres/Oxigraph stack, give each a distinct `topic_prefix` (e.g. `fbf/hospital-a/...` vs `fbf/hospital-b/...`), and their points/equipment never collide — no migration, no new column, because identity was never global to begin with. For campus-level filtering ("all faults across sites," "just Site B"), tag each equip with a `site` Haystack tag through the ingestion path that already exists (`ingest_tags`/`ingest_equip_tags` in `ingest.py` take an unconstrained tag dict). `site` is then just a queryable property in Oxigraph like any other tag — no new mechanism. This only holds if sites can share one Postgres/Oxigraph. If a site must stay data-sovereign or run standalone when disconnected, it needs its own full stack, with the central campus stack pulling from it — the same pull shape `haystack_puller`/`sql_puller` already implement, just pointed at a peer Baler instead of a third-party source. That's a bigger decision than this page covers and hasn't come up yet. ## Who sees which data Each site can own one or more topic prefixes (`POST /auth/sites` with `topic_prefixes`, or `PUT /auth/sites/{id}/topic-prefixes`). A site owning `fbf/hospital-a` owns every point `urn:point:fbf/hospital-a/...` and every equip `urn:equip:fbf/hospital-a/...`. - **No site anywhere owns a prefix:** everyone sees everything. This is a single-tenant install, and it's how every install behaved before scoping existed. - **Your org owns prefixes:** you see only those - your token's `site_ids`' prefixes, or all of your org's if the token carries none. - **Your org owns none** (while another does): you see everything *except* what other orgs claimed. Scoping is per org, so one org partitioning its data doesn't blind the rest. - **Locations** (Site, Building, Floor, Zone) have no prefix: one is visible when it contains something you can see (through `hasPart`, backwards along `feeds`, `hasPoint`), and hidden when it contains something another org claimed. - Out-of-scope records read as missing, never as "forbidden", so their existence isn't revealed. Graph reads apply scope **inside the query**: to the result and to every record a filter path reaches, so `spaceRef->dis == "x"` can't test tags on a record you can't see. This applies to the Haystack API (`/api`: read, nav, hisRead, hisWrite, watches) and to `/ingest/his` and `/ingest/history`. - **Prefixes never overlap across orgs.** Claiming `fbf` while another org owns `fbf/hospital-a` (or the reverse) is rejected, otherwise the broader claim would expose the other org's data. - **First claim wins.** In a shared instance, an org can claim a prefix no one owns yet, including one another tenant's devices already publish under. Provision each site's prefixes when you create the site. Also scoped, by the same rule: - **Faults** (`/fault/faults`): the list shows only faults on your points; acking or snoozing another site's fault is a 404. - **Derivations**: a derivation's targets list, dry-run trace and target enable cover only equipment in your sites. - **Validation** (`/validate/validate`): only violations on your points and equipment, and `conforms` describes those alone. Shared across the org (configuration, not site data): fault rules, webhooks and derivation definitions. Known gap: a derivation's output points (`urn:point:computed/...`) belong to no site prefix, so once scoping is on they don't appear in `/api` or `/ingest/his` reads. Read them unscoped (a single-tenant install) until computed points are scoped through the equipment they attach to. ## Why not a mesh network Some building platforms connect sites as a peer-to-peer mesh: every node has its own identity, and any node can route requests (reads, writes, history, watches) to any other. That's the right shape when sites need to talk to each other. Baler's topology is a star, not a mesh: N sites report up to one gateway/broker/store, and sites never need to talk to each other. A star doesn't need peer identity, routing, or an RPC surface between leaves; a plain broker's client/hub model already covers everything a star topology requires. Building mesh machinery (node keypairs, signed frames, a node-type registry) for a topology that's provably a star would be solving a problem this system doesn't have. The property that does matter is that the site opens the connection outbound, so it doesn't fight building-network firewalls. MQTT already has that property. Once an MQTT client opens its session to the broker, either side can send over it, and the broker never needs a route back into the site's network. Infrastructure already in this stack covers the one topology this system actually has. ## Reaching back into a site: commands, not just ingest Windrow already has a real REST surface for exactly the kind of thing you'd want to trigger remotely — `POST /discover`, `POST /learn`, `POST /modbus/discover` (takes a CIDR), device/credential management (see Windrow's own `docs/bacnet-discovery-api.md` and `docs/periodic-discovery-and-credentials.md`). The gap isn't capability, it's reachability: that server binds inside the building, and a central Baler has no inbound path to it, especially on a network that can only originate connections outward and never receive them. **Implemented (2026-09-06):** `windrow.api` subscribes to a command topic — `cmd/{site}/{instance}/#` — on the same MQTT connection it already holds open to publish readings, once started with `--site-id` (see Windrow's own `docs/command-channel.md`). A command message (e.g. `{"action": "modbus_discover", "cidr": "10.2.0.0/24"}`) gets dispatched in-process to the same `handle_discover`/`handle_learn`/ `handle_modbus_discover` functions its own HTTP routes call, and the outcome is published to `cmd/{site}/{instance}/result`. No new server, no new port, no firewall exception — it rides the identical egress rule that already lets Windrow publish telemetry today, because it's the same connection. The two things this needed to be safe once it controls real scans, not just telemetry, are both done: - **Persistent session** (`clean_session=False`, QoS 1) on Windrow's client, set once `--site-id` is given (`fbf/src/windrow/mqtt_sink.py`'s `connect()`) — a command sent while the site is offline is queued by the broker instead of dropped. - **Per-site broker ACLs** (`mosquitto/acl`). `allow_anonymous` is off; every client authenticates, and a `pattern readwrite cmd/%u/#` ACL entry means a site's username *is* its command-topic scope — a client can only read/write its own `cmd//#`, enforced by the broker itself, not by convention. Live-verified: an authenticated `site-a` client publishing to `cmd/site-b/...` is silently dropped by the broker — a subscriber on `cmd/site-b/#` never receives it. ## Edge nodes: per-site broker accounts A Windrow running *at* a site (a Raspberry Pi on the building network) connects to this broker from outside the Docker host. It gets its own account, made in **Ops → Edge nodes** (admin-only; API: `POST /auth/broker-accounts`, see the auth API's OpenAPI spec): - **One account per node, tied to one site.** It may publish only under that site's topic prefixes (`` and `/…`; a prefix is a whole topic level, so `fbf/hospital-a` doesn't cover `fbf/hospital-ab`), and publish/subscribe only on its own `cmd//#`. It can't read telemetry at all, not even its own site's. The broker enforces this (mosquitto's dynamic-security plugin), so one site's node can't write into another site's points. - **Use the Windrow's `--site-id` as the username** - Windrow's command channel is `cmd/{site-id}/{instance}/#`, which is the account's `cmd//#`. Its `--topic-prefix` must sit under the site's prefix. - **The password is generated and shown once**, with where to connect (`BALER_MQTT_PUBLIC_HOST`/`_PORT`/`_TLS_PORT` in `.env`). Baler stores only the account's metadata; the broker keeps a salted hash. **New password** rotates it (the old one stops working at once); **Revoke** deletes the account and disconnects the node. - **Changing a site's topic prefixes re-applies its nodes' permissions.** If the broker is unreachable at that moment the prefixes are still saved and the response says so (`broker_error`); the next change retries. **Use TLS (port 8883) from outside the host.** Set `BALER_MQTT_TLS_DIR` to a directory holding the certificate (`cert.pem` + `key.pem`, or a Caddy-style `.crt` + `.key`) and mosquitto also listens on 8883 with TLS, authenticating the same accounts. On nash-srv, `setup-runner.sh` step 7 points it at the Let's Encrypt cert Caddy already keeps for `timberdoodle.nash.engineering` and installs a systemd path unit that restarts mosquitto when Caddy renews it; 8883 is published publicly there while plain 1883 stays loopback-only. A node connects with `--mqtt-host timberdoodle.nash.engineering --mqtt-port 8883` and TLS on, verifying against the system CA store (it's a public Let's Encrypt cert). **On a Raspberry Pi:** Windrow's `deploy/pi/install-edge.sh` sets a Pi up as a site gateway (Windrow itself), a simulated BACnet or Modbus device, or a simulated weather station, each under systemd with the edge account's credentials in `/etc/windrow/edge.env`. See Windrow's [docs/edge-nodes.md](https://github.com/jacobnash/Windrow/blob/main/docs/edge-nodes.md), and [Edge nodes](/edge-nodes) for the runbook: which topology to use, connecting the devices, and checking them end to end. How it fits with the existing accounts: `MOSQUITTO_USERS` (the `baler` backend account and any hand-added ones) stay in the password/ACL files. The plugin's own file, `dynamic-security.json` on the `mosquitto-data` volume, holds only edge accounts; a client it doesn't know falls through to the files. The `baler` account manages the plugin over `$CONTROL/dynamic-security/v1` (granted in `mosquitto/acl`), so there's no separate broker admin secret. Back up the `mosquitto-data` volume with the rest: losing that file loses the edge accounts (re-create them; Ops still lists them, and revoking tolerates an account the broker no longer has). ## MQTT delivery guarantees This matters more once sites are remote and links aren't as reliable as one Docker network. This section used to describe several open gaps; as of 2026-09-06 all of them are closed. QoS 1 publish and disk spooling on Windrow's side, and `clean_session=False` on `mqtt_listener.py`, turned out to already be done by the time this page was checked against the code again — the mosquitto queue cap below was the one piece still genuinely open: - `fbf/src/windrow/mqtt_sink.py`'s `publish_reading()` publishes at QoS 1, and spools to a local JSONL file (replayed on reconnect) when the client isn't currently connected — a link drop no longer means silently dropped readings on Windrow's side. - `mqtt_listener.py`'s client uses a stable `client_id` and `clean_session=False`, so mosquitto's queued messages actually have a session to be queued against. `fault_detector.py` and `derivation_engine.py` authenticate the same way (all three now go through the shared `baler.mqtt_util.make_client()`) but still use ephemeral sessions — they evaluate live telemetry, not a durable log, so losing a queued backlog across a restart is an accepted gap, unlike the command channel below where a dropped message means a scan silently never runs. - `mosquitto` now runs with a real mounted config (`mosquitto/mosquitto.conf`) setting `persistence true` and `max_queued_messages 200000` (up from the image default of 1000; 10000 until the scale test (#190) showed a listener restart overflowing it at 5k points) — see `mosquitto/entrypoint.sh` for how the password file is generated from `.env` at container start rather than committing hashed credentials. - `retain=True` is still not a history buffer — it only caches the latest value per topic for new subscribers. The queue depth above is what actually protects an outage window, not `retain`. 200000 queued messages per client is still not unlimited — a busy multi-point site can still overflow it during a long enough outage. If that turns out to matter in practice, the next step is backfilling from BACnet/Modbus devices' own trend logs on reconnect rather than raising the cap indefinitely. ## When mesh-style networking would actually be justified If Baler deployments themselves needed to federate — separate orgs, data-sovereignty requirements forcing separate stores — *and* those deployments needed to query or command each other live, not just roll up for viewing, that's when a peer-to-peer mesh starts to earn its cost. That's a stronger requirement than campus aggregation and isn't implied by anything on the roadmap today; if it shows up, revisit this page and design that layer openly (on the Haystack protocol), rather than growing one by accident.