# Architecture

Baler is fourteen containers under one `docker-compose.yml`: five
infrastructure services, eight daemons (one shared Docker image, a
different `command:` each), and an nginx gateway in front of the five
HTTP APIs (ingest, fault, derivation, validate, auth). `docker compose up
-d` starts all of it. Nothing here is guessed — this is the real
`docker-compose.yml` and the real CLI entry points.

A 15th container, `haystack_puller`, exists but is opt-in — gated behind
`docker compose --profile haystack-pull`, so plain `docker compose up -d`
correctly excludes it. See [Haystack Puller](/haystack-puller).

### Ingest path

Two front doors (BACnet/Modbus over MQTT, or a direct HTTP push), one
shared write path into both stores:

<Mermaid chart={`graph LR
    subgraph Sources
        Windrow["Windrow (BACnet/Modbus)"]
        PUSH["push-style sources"]
    end

    subgraph Transport
        MQ[(mosquitto)]
        GW["gateway (nginx :8080)"]
    end

    subgraph Processors
        ML["mqtt_listener"]
        IA["ingest_api :8000"]
    end

    subgraph Stores
        OX[(oxigraph)]
        PG[(postgres/timescale)]
    end

    Windrow -->|"MQTT fbf/#"| MQ
    PUSH -->|"POST /ingest/ingest"| GW
    MQ -->|"subscribes fbf/#"| ML
    GW --> IA
    ML --> OX
    ML --> PG
    IA --> OX
    IA --> PG
`} />

### Fault pipeline

`fault_api.py` and `fault_detector.py` are decoupled on purpose (see
[Fault Detection](/fault-detection)) — they share nothing but
`rules.json`/`webhooks.json`, drawn here as the one node both connect to,
not as a direct edge between the two processes:

<Mermaid chart={`graph LR
    GW["gateway (nginx :8080)"] --> FA["fault_api :8002"]
    FA -->|CRUD| FCFG[("rules.json /\\nwebhooks.json")]
    FA -->|"GET /faults"| PG[(postgres/timescale)]

    MQ[(mosquitto)] -->|"Cur mode"| FD["fault_detector"]
    FD -->|reads| FCFG
    FD -->|"His mode, periodic sweep"| PG
    FD -->|"fault open/resolve"| WH(("webhook URLs"))
`} />

### Derivation pipeline

Same shape as the fault pipeline, one config file apart: `derivation_api.py`
only ever writes `derivations.json`; `derivation_engine.py` is the only
process that evaluates one:

<Mermaid chart={`graph LR
    GW["gateway (nginx :8080)"] --> DA["derivation_api :8003"]
    DA -->|CRUD| DCFG[("derivations.json")]

    MQ[(mosquitto)] -->|"cur-mode"| DE["derivation_engine"]
    DE -->|reads| DCFG
    DE -->|"sweep + rollups"| PG[(postgres/timescale)]
    DE -->|"lineage triples"| OX[(oxigraph)]
`} />

## The gateway

An `nginx:alpine` container (`gateway/nginx.conf`) is the one thing
external callers talk to — `localhost:8080`. It does two jobs:

- **Path routing**, so callers don't need to know five separate ports:
  `/ingest/*` → `ingest_api:8000`, `/fault/*` → `fault_api:8002`,
  `/derivation/*` → `derivation_api:8003`, `/validate/*` →
  `validate_api:8005`, `/auth/*` → `auth_api:8006` (plus the exact-match
  `/auth/login` and `/auth/orgs` locations, since those two routes are
  public — no token exists yet at login time, and creating the first org
  can't require one either). It also serves the static demo UI at `/ui/`
  directly (an nginx `alias` onto `./ui/`, not a sixth process). None of
  the five API containers publish host ports at all — the gateway is the
  only way in from outside the Docker network.
- **JWT auth, enforced in nginx itself** via njs (`gateway/njs/`): every
  request is verified against a per-route policy table
  (`gateway/njs/policy.js`) before it reaches a backend — no per-backend
  Python auth code. See [Gateway Auth](/gateway-auth) for how to create
  an org/admin and get a token.

This is deliberately *not* a merge of the five API processes into one
codebase — they stay separate builds/images/restarts, same as before,
consistent with the decoupling described throughout this page. The
gateway only changes how they're reached from outside.

One caveat worth knowing: `proxy_pass` targets resolve once at nginx
startup and stay cached. If a single backend container is recreated
independently (not the whole stack via `docker compose up -d`), restart
the gateway too (`docker compose restart gateway`) so it picks up the new
address — see the `ponytail:` comment in `gateway/nginx.conf`.

The gateway's auth is a checkpoint at the network edge, not a fix for
what's behind it — the derivation API's code-exec sandbox is still only
defense in depth (see below), and `fault_api.py`'s webhook URLs still
have [no SSRF protection](/webhooks#no-ssrf-protection-in-v1). Both
warnings stand exactly as documented; the gateway just means something
external now has to authenticate before it can reach either.

## Infrastructure (docker-compose)

| Service | Image | Purpose | Port(s) |
|---|---|---|---|
| `mosquitto` | `eclipse-mosquitto:2` | MQTT broker every source and every consumer shares. | 1883 |
| `oxigraph` | `ghcr.io/oxigraph/oxigraph:0.5.13` | RDF/SPARQL graph store — entity identity, tags, relationships. | 7878 |
| `postgres` | `timescale/timescaledb:2.30.0-pg16` | Timescale hypertable, `point_history` — the actual reading values. | 5433 → 5432 in-container |
| `jaeger` | `jaegertracing/jaeger:2.23.0` (v2; config in `jaeger/config.yaml`) | Trace collection/UI. Every process below exports here. | 16686 (UI), 4318 (OTLP HTTP) |
| `grafana` | `grafana/grafana-oss:13.0.2` | Dashboards over Postgres — a datasource and starter dashboard are auto-provisioned. Login is the image default `admin`/`admin`; anonymous admin access is available for local dev via `GRAFANA_ANON_ENABLED=true` but off by default. | 3033 → 3000 in-container |
| `gateway` | `nginx:1.31.8-alpine` | JWT-auth-gated path router in front of the five HTTP APIs, plus the static UI. | 8080 |

Postgres is deliberately remapped to host port 5433 rather than the
default 5432, to avoid colliding with a locally-running Postgres
instance — easy to miss if you're connecting a client by hand. Its
password comes from `POSTGRES_PASSWORD` in `.env` (falls back to
`baler` if unset) rather than being hardcoded in
`docker-compose.yml`.

## Why two stores, not one

Time-series values and graph/entity data have genuinely different access
patterns, so they live in genuinely different systems rather than one
database wearing two hats:

- **Postgres/TimescaleDB** for `point_history` — because the natural
  consumer of a time-series (Grafana, or literally anything that speaks
  SQL) should be able to read it with zero custom connector or plugin.
- **Oxigraph** for entity identity, Haystack tags, and Brick
  relationships — because the graph doesn't know or care which ontology
  (Brick, Haystack, DBO) it's loaded with. That's a load-time choice
  (`store.load_ontology`), not a schema baked into the code — the whole
  point of being "ontology-agnostic."

**`load_ontology` is proven, but no daemon calls it automatically.**
`store.py`/`remote_store.py` both implement it and it's exercised by
`tests/test_store.py`, `test_remote_store.py`, `test_mapping.py`,
`test_equip_mapping.py` (all against the tiny `tests/fixtures/brick_subset.ttl`).
Nothing in `docker-compose.yml`'s startup path loads a full Brick ontology
into the running Oxigraph on its own. That's not a bug: `mapping.classify_point`
asserts `rdf:type brick:<Class>` triples directly and never queries an
existing class hierarchy to validate them, so the ingest/classify pipeline
works fine with an ontology-free graph. `validate_api.py`'s SHACL check
(see [SHACL validation](/validate)) loads its own private copy of the same
file for shape-checking and is unaffected either way. Loading a real
ontology only matters if you want SPARQL queries that walk
`rdfs:subClassOf` against the live graph. Run `python ontology_demo.py`
yourself to do it — it loads the vendored `ontology/Brick-only.ttl`
(no external download needed) into `RemoteStore`'s dedicated ontology
graph via `load_ontology`, then runs a `subClassOf*` query against the
live store to prove it landed. The one feature that needs Brick's
hierarchy at query time — [the `readAll(...).hisRead(...)` filter](/his-query),
where `Temperature_Sensor` must find every subclass — deliberately does
*not* depend on this: `brick_vocab.py` reads the vendored
`ontology/Brick-only.ttl` directly and hands SPARQL a bound class list,
both because the ontology may not be loaded and because Oxigraph takes
tens of seconds on an unbound `rdfs:subClassOf*` inside a filter when
it is.

## The eight daemons

All eight now build from one root `Dockerfile` (a single `pip install -e .`
image, editable so the `__file__`-relative paths each API uses to find
its own OpenAPI spec keep resolving the same way they do outside Docker)
and run as `docker-compose.yml` services, each with the same image and a
different `command:`. They reach the infra containers above by
Docker-network service name rather than `localhost`, via three env vars
(`BALER_PG_DSN`, `OXIGRAPH_URL`, `MQTT_HOST` — see the
`x-daemon-env` anchor in `docker-compose.yml`), each read with a
`localhost`-pointing fallback in the code
(`timeseries.DEFAULT_DSN`, `RemoteStore.__init__`, and the `--mqtt-host`/
`--oxigraph-url` CLI flag defaults) — so nothing changes for anyone still
running a process bare-metal outside Docker.

| Process | Reads from | Writes to | Port |
|---|---|---|---|
| `mqtt_listener.py` | mosquitto (`fbf/#`) | oxigraph, postgres | — |
| `ingest_api.py` | HTTP `POST /ingest` | oxigraph, postgres | 8000 |
| `fault_api.py` | HTTP CRUD | `rules.json`/`webhooks.json`, postgres (`GET /faults`) | 8002 |
| `fault_detector.py` | mosquitto (Cur), postgres (His, periodic sweep) | postgres (`faults` table), outbound webhooks | — |
| `derivation_api.py` | HTTP CRUD, `POST /derivations/test`, `/dry-run` | `derivations.json`, postgres (health) | 8003 |
| `derivation_engine.py` | mosquitto (cur-mode), postgres (sweep, His inputs), oxigraph (SPARQL select) | postgres (`point_history`), oxigraph (lineage triples) | — |
| `validate_api.py` | HTTP `POST /validate` (no body — always validates current live state) | oxigraph (read-only, via `RemoteStore`) | 8005 |
| `auth_api.py` | HTTP CRUD (`/login`, `/orgs`, `/users`, `/sites`, `/api-keys`, `/logout`), gateway-internal `GET /internal/revoked-jtis` | postgres (orgs/sites/users/API keys/revoked JTIs) | 8006 |
| `commissioning_api.py` | HTTP (`/cx/`): projects, phases, inventory, runs, issues and risks, procedures, spec and alignment, active tests, the agent - see `todo/commissioning-plan.md` | postgres (`cx_*` tables; writes `faults` when a project mirrors, `point_write_levels` in active tests, `record_edits` on spec write-back), oxigraph (scoped SPARQL; `record_edit` write-back), `point_history` (read), MQTT (`point_write`, active tests only), an LLM for spec PDFs and the agent - Claude or any OpenAI-compatible endpoint, `BALER_LLM_PROVIDER`, see [Commissioning models](/commissioning-llm) | 8009 |

Each daemon's container bind-mounts the repo root (`.:/app`) so
`rules.json`/`webhooks.json`/`derivations.json` — read/written by these
processes at their default relative paths — are the same files on disk
whether a process is running in Docker or bare-metal; there's no separate
in-container copy to keep in sync. Postgres startup is the one real
ordering hazard (a daemon connecting before Postgres accepts connections
would just crash), so `postgres` has a `pg_isready` healthcheck and every
daemon `depends_on` it with `condition: service_healthy`; oxigraph and
mosquitto start fast enough in practice not to need the same treatment.

Plus two one-shot tools that aren't daemons at all: `autotag.py` (rules,
then the [model step](/model-step) on what they leave; what isn't settled
becomes a mapping proposal for a person, and the rule files are never
edited) and the static `ui/` — served both by the gateway at
`/ui/` (no separate process) and, standalone, by
`python -m baler.ui_server` on `:8004` for anyone not going
through Docker. Either way it talks to `derivation_api.py`/`fault_api.py`
over HTTP from the browser; through the gateway that's same-origin
(`/ingest`, `/fault`, `/derivation`). Two of `ui/`'s pages are thin
browser clients over one of those APIs each and nothing more (no new
expression language, no server-side logic beyond what the API already
has): `derivations.html` is a form over `derivation_api.py`'s existing
`POST /derivations/test`/`/dry-run`/`/derivations` (write `fn_source`,
run it, save it), and its sibling `alarms.html` is a table over
`fault_api.py`'s `GET /faults` plus `POST /faults/{id}/ack`/`/snooze` —
see [Fault Detection](/fault-detection) for that one. Since the gateway
auth migration,
though, that's a Bearer token, not basic auth — there's no
browser-native credential prompt for a JWT. `ui/app.js` has to hold a
login gate itself: exchange credentials for a token against
`POST /auth/login`, keep the token client-side, and attach
`Authorization: Bearer <token>` to every fetch it makes through the
gateway. See [Gateway Auth](/gateway-auth) for the full login flow and
token shape.

`mqtt_listener.py` and `ingest_api.py` are two front doors onto the same
ingest code — a source can push over HTTP or publish over MQTT and it
behaves identically, because point identity is just the topic string
(`urn:point:{topic}`). The listener writes readings in batches
(`ingest_readings()`, the batched form of `ingest_reading()`): `on_message`
queues a reading, a writer thread commits up to 5,000 per transaction (a
`COPY` into a staging table, then one upsert) at least every 250 ms, and
only then acknowledges their MQTT messages. Nothing is acknowledged before
it's durably committed, so a crash of the listener or Postgres leaves the
unacknowledged readings with the broker, which redelivers them; the writes
are idempotent on `(point, ts)`. Tags and equip messages are still handled
one at a time, as they arrive. The queue is bounded: when it's full, the
backlog waits at the broker (`max_inflight_messages`, `max_queued_messages`
in `mosquitto/mosquitto.conf`). The [scale test](/scale-test) measures how
far this goes.

<Mermaid chart={`sequenceDiagram
    participant Src as Source (MQTT or push client)
    participant ML as mqtt_listener.py
    participant IA as ingest_api.py
    participant Ing as ingest.ingest_reading()
    participant OX as oxigraph
    participant PG as postgres

    alt over MQTT
        Src->>ML: publish fbf/# envelope
        ML->>Ing: ingest_readings(batch), then ack the batch
    else over HTTP
        Src->>IA: POST /ingest
        IA->>Ing: ingest_reading(envelope)
    end
    Ing->>OX: link point/equip triples
    Ing->>PG: write point_history row
`} />

`fault_api.py` and `fault_detector.py` are decoupled on purpose:
`fault_api.py` only ever writes `rules.json`/`webhooks.json` and reads
`faults` from Postgres; `fault_detector.py` is the only process that
actually evaluates rules. They share nothing but those two config files —
see [Fault Detection](/fault-detection) for why. Containerizing both
didn't change that — they're still two separate services/images/restarts
in `docker-compose.yml`, just no longer manually started.

`derivation_api.py` and `derivation_engine.py` follow the identical
shape: the API only ever writes `derivations.json` (plus per-target
health in Postgres) and the engine is the only process that evaluates
derivations, sharing nothing but that one config file.

## Why stdlib `http.server`, not a framework

`ingest_api.py`, `fault_api.py`, `derivation_api.py`, `validate_api.py`,
and `auth_api.py` are all built on Python's stdlib
`http.server.ThreadingHTTPServer` rather than Flask/FastAPI/etc. None of
them is more than a handful of routes — a framework dependency for that
is a cost with no payoff here. Each request
gets its own Postgres connection from a pool
(`timeseries.connect_pool()`), not one shared connection across threads —
`psycopg.Connection` isn't safe for concurrent multi-thread use, and a
shared connection previously caused a silent, unrecoverable-without-restart
failure in `ingest_api.py`.

## Tracing

Every process calls `tracing.init_tracing(<service_name>)` on startup and
gets a real OpenTelemetry span around every nontrivial operation — not a
log line. This was a direct response to a multi-hour debugging session
where a silent failure had zero visible signal. All eight daemons export
to Jaeger's OTLP HTTP endpoint — `jaeger:4318` inside Docker,
`localhost:4318` outside it, both by the same
`OTEL_EXPORTER_OTLP_ENDPOINT` override mechanism; view traces at
`localhost:16686`.