Copy page

Documentation

Architecture

Baler is fourteen containers under one docker-compose.yml: five infrastructure services, eight daemons (one shared Docker image, a different command: each), and an nginx gateway in front of the five HTTP APIs (ingest, fault, derivation, validate, auth). docker compose up -d starts all of it. Nothing here is guessed — this is the real docker-compose.yml and the real CLI entry points.

A 15th container, haystack_puller, exists but is opt-in — gated behind docker compose --profile haystack-pull, so plain docker compose up -d correctly excludes it. See Haystack Puller.

Ingest path

Two front doors (BACnet/Modbus over MQTT, or a direct HTTP push), one shared write path into both stores:

Fault pipeline

fault_api.py and fault_detector.py are decoupled on purpose (see Fault Detection) — they share nothing but rules.json/webhooks.json, drawn here as the one node both connect to, not as a direct edge between the two processes:

Derivation pipeline

Same shape as the fault pipeline, one config file apart: derivation_api.py only ever writes derivations.json; derivation_engine.py is the only process that evaluates one:

The gateway

An nginx:alpine container (gateway/nginx.conf) is the one thing external callers talk to — localhost:8080. It does two jobs:

• Path routing, so callers don't need to know five separate ports: /ingest/* → ingest_api:8000, /fault/* → fault_api:8002, /derivation/* → derivation_api:8003, /validate/* → validate_api:8005, /auth/* → auth_api:8006 (plus the exact-match /auth/login and /auth/orgs locations, since those two routes are public — no token exists yet at login time, and creating the first org can't require one either). It also serves the static demo UI at /ui/ directly (an nginx alias onto ./ui/, not a sixth process). None of the five API containers publish host ports at all — the gateway is the only way in from outside the Docker network.
• JWT auth, enforced in nginx itself via njs (gateway/njs/): every request is verified against a per-route policy table (gateway/njs/policy.js) before it reaches a backend — no per-backend Python auth code. See Gateway Auth for how to create an org/admin and get a token.

This is deliberately not a merge of the five API processes into one codebase — they stay separate builds/images/restarts, same as before, consistent with the decoupling described throughout this page. The gateway only changes how they're reached from outside.

One caveat worth knowing: proxy_pass targets resolve once at nginx startup and stay cached. If a single backend container is recreated independently (not the whole stack via docker compose up -d), restart the gateway too (docker compose restart gateway) so it picks up the new address — see the ponytail: comment in gateway/nginx.conf.

The gateway's auth is a checkpoint at the network edge, not a fix for what's behind it — the derivation API's code-exec sandbox is still only defense in depth (see below), and fault_api.py's webhook URLs still have no SSRF protection. Both warnings stand exactly as documented; the gateway just means something external now has to authenticate before it can reach either.

Infrastructure (docker-compose)

Service	Image	Purpose	Port(s)
mosquitto	eclipse-mosquitto:2	MQTT broker every source and every consumer shares.	1883
oxigraph	ghcr.io/oxigraph/oxigraph:0.5.13	RDF/SPARQL graph store — entity identity, tags, relationships.	7878
postgres	timescale/timescaledb:2.30.0-pg16	Timescale hypertable, point_history — the actual reading values.	5433 → 5432 in-container
jaeger	jaegertracing/jaeger:2.23.0 (v2; config in jaeger/config.yaml)	Trace collection/UI. Every process below exports here.	16686 (UI), 4318 (OTLP HTTP)
grafana	grafana/grafana-oss:13.0.2	Dashboards over Postgres — a datasource and starter dashboard are auto-provisioned. Login is the image default admin/admin; anonymous admin access is available for local dev via GRAFANA_ANON_ENABLED=true but off by default.	3033 → 3000 in-container
gateway	nginx:1.31.8-alpine	JWT-auth-gated path router in front of the five HTTP APIs, plus the static UI.	8080

Postgres is deliberately remapped to host port 5433 rather than the default 5432, to avoid colliding with a locally-running Postgres instance — easy to miss if you're connecting a client by hand. Its password comes from POSTGRES_PASSWORD in .env (falls back to baler if unset) rather than being hardcoded in docker-compose.yml.

Why two stores, not one

Time-series values and graph/entity data have genuinely different access patterns, so they live in genuinely different systems rather than one database wearing two hats:

• Postgres/TimescaleDB for point_history — because the natural consumer of a time-series (Grafana, or literally anything that speaks SQL) should be able to read it with zero custom connector or plugin.
• Oxigraph for entity identity, Haystack tags, and Brick relationships — because the graph doesn't know or care which ontology (Brick, Haystack, DBO) it's loaded with. That's a load-time choice (store.load_ontology), not a schema baked into the code — the whole point of being "ontology-agnostic."

load_ontology is proven, but no daemon calls it automatically. store.py/remote_store.py both implement it and it's exercised by tests/test_store.py, test_remote_store.py, test_mapping.py, test_equip_mapping.py (all against the tiny tests/fixtures/brick_subset.ttl). Nothing in docker-compose.yml's startup path loads a full Brick ontology into the running Oxigraph on its own. That's not a bug: mapping.classify_point asserts rdf:type brick:<Class> triples directly and never queries an existing class hierarchy to validate them, so the ingest/classify pipeline works fine with an ontology-free graph. validate_api.py's SHACL check (see SHACL validation) loads its own private copy of the same file for shape-checking and is unaffected either way. Loading a real ontology only matters if you want SPARQL queries that walk rdfs:subClassOf against the live graph. Run python ontology_demo.py yourself to do it — it loads the vendored ontology/Brick-only.ttl (no external download needed) into RemoteStore's dedicated ontology graph via load_ontology, then runs a subClassOf* query against the live store to prove it landed. The one feature that needs Brick's hierarchy at query time — the readAll(...).hisRead(...) filter, where Temperature_Sensor must find every subclass — deliberately does not depend on this: brick_vocab.py reads the vendored ontology/Brick-only.ttl directly and hands SPARQL a bound class list, both because the ontology may not be loaded and because Oxigraph takes tens of seconds on an unbound rdfs:subClassOf* inside a filter when it is.

The eight daemons

All eight now build from one root Dockerfile (a single pip install -e . image, editable so the __file__-relative paths each API uses to find its own OpenAPI spec keep resolving the same way they do outside Docker) and run as docker-compose.yml services, each with the same image and a different command:. They reach the infra containers above by Docker-network service name rather than localhost, via three env vars (BALER_PG_DSN, OXIGRAPH_URL, MQTT_HOST — see the x-daemon-env anchor in docker-compose.yml), each read with a localhost-pointing fallback in the code (timeseries.DEFAULT_DSN, RemoteStore.__init__, and the --mqtt-host/ --oxigraph-url CLI flag defaults) — so nothing changes for anyone still running a process bare-metal outside Docker.

Process	Reads from	Writes to	Port
mqtt_listener.py	mosquitto (fbf/#)	oxigraph, postgres	—
ingest_api.py	HTTP POST /ingest	oxigraph, postgres	8000
fault_api.py	HTTP CRUD	rules.json/webhooks.json, postgres (GET /faults)	8002
fault_detector.py	mosquitto (Cur), postgres (His, periodic sweep)	postgres (faults table), outbound webhooks	—
derivation_api.py	HTTP CRUD, POST /derivations/test, /dry-run	derivations.json, postgres (health)	8003
derivation_engine.py	mosquitto (cur-mode), postgres (sweep, His inputs), oxigraph (SPARQL select)	postgres (point_history), oxigraph (lineage triples)	—
validate_api.py	HTTP POST /validate (no body — always validates current live state)	oxigraph (read-only, via RemoteStore)	8005
auth_api.py	HTTP CRUD (/login, /orgs, /users, /sites, /api-keys, /logout), gateway-internal GET /internal/revoked-jtis	postgres (orgs/sites/users/API keys/revoked JTIs)	8006
commissioning_api.py	HTTP (/cx/): projects, phases, inventory, runs, issues and risks, procedures, spec and alignment, active tests, the agent - see todo/commissioning-plan.md	postgres (cx_* tables; writes faults when a project mirrors, point_write_levels in active tests, record_edits on spec write-back), oxigraph (scoped SPARQL; record_edit write-back), point_history (read), MQTT (point_write, active tests only), an LLM for spec PDFs and the agent - Claude or any OpenAI-compatible endpoint, BALER_LLM_PROVIDER, see Commissioning models	8009

Each daemon's container bind-mounts the repo root (.:/app) so rules.json/webhooks.json/derivations.json — read/written by these processes at their default relative paths — are the same files on disk whether a process is running in Docker or bare-metal; there's no separate in-container copy to keep in sync. Postgres startup is the one real ordering hazard (a daemon connecting before Postgres accepts connections would just crash), so postgres has a pg_isready healthcheck and every daemon depends_on it with condition: service_healthy; oxigraph and mosquitto start fast enough in practice not to need the same treatment.

Plus two one-shot tools that aren't daemons at all: autotag.py (rules, then the model step on what they leave; what isn't settled becomes a mapping proposal for a person, and the rule files are never edited) and the static ui/ — served both by the gateway at /ui/ (no separate process) and, standalone, by python -m baler.ui_server on :8004 for anyone not going through Docker. Either way it talks to derivation_api.py/fault_api.py over HTTP from the browser; through the gateway that's same-origin (/ingest, /fault, /derivation). Two of ui/'s pages are thin browser clients over one of those APIs each and nothing more (no new expression language, no server-side logic beyond what the API already has): derivations.html is a form over derivation_api.py's existing POST /derivations/test//dry-run//derivations (write fn_source, run it, save it), and its sibling alarms.html is a table over fault_api.py's GET /faults plus POST /faults/{id}/ack//snooze — see Fault Detection for that one. Since the gateway auth migration, though, that's a Bearer token, not basic auth — there's no browser-native credential prompt for a JWT. ui/app.js has to hold a login gate itself: exchange credentials for a token against POST /auth/login, keep the token client-side, and attach Authorization: Bearer <token> to every fetch it makes through the gateway. See Gateway Auth for the full login flow and token shape.

mqtt_listener.py and ingest_api.py are two front doors onto the same ingest code — a source can push over HTTP or publish over MQTT and it behaves identically, because point identity is just the topic string (urn:point:{topic}). The listener writes readings in batches (ingest_readings(), the batched form of ingest_reading()): on_message queues a reading, a writer thread commits up to 5,000 per transaction (a COPY into a staging table, then one upsert) at least every 250 ms, and only then acknowledges their MQTT messages. Nothing is acknowledged before it's durably committed, so a crash of the listener or Postgres leaves the unacknowledged readings with the broker, which redelivers them; the writes are idempotent on (point, ts). Tags and equip messages are still handled one at a time, as they arrive. The queue is bounded: when it's full, the backlog waits at the broker (max_inflight_messages, max_queued_messages in mosquitto/mosquitto.conf). The scale test measures how far this goes.

fault_api.py and fault_detector.py are decoupled on purpose: fault_api.py only ever writes rules.json/webhooks.json and reads faults from Postgres; fault_detector.py is the only process that actually evaluates rules. They share nothing but those two config files — see Fault Detection for why. Containerizing both didn't change that — they're still two separate services/images/restarts in docker-compose.yml, just no longer manually started.

derivation_api.py and derivation_engine.py follow the identical shape: the API only ever writes derivations.json (plus per-target health in Postgres) and the engine is the only process that evaluates derivations, sharing nothing but that one config file.

Why stdlib http.server, not a framework

ingest_api.py, fault_api.py, derivation_api.py, validate_api.py, and auth_api.py are all built on Python's stdlib http.server.ThreadingHTTPServer rather than Flask/FastAPI/etc. None of them is more than a handful of routes — a framework dependency for that is a cost with no payoff here. Each request gets its own Postgres connection from a pool (timeseries.connect_pool()), not one shared connection across threads — psycopg.Connection isn't safe for concurrent multi-thread use, and a shared connection previously caused a silent, unrecoverable-without-restart failure in ingest_api.py.

Tracing

Every process calls tracing.init_tracing(<service_name>) on startup and gets a real OpenTelemetry span around every nontrivial operation — not a log line. This was a direct response to a multi-hour debugging session where a silent failure had zero visible signal. All eight daemons export to Jaeger's OTLP HTTP endpoint — jaeger:4318 inside Docker, localhost:4318 outside it, both by the same OTEL_EXPORTER_OTLP_ENDPOINT override mechanism; view traces at localhost:16686.