Copy page

Documentation

Haystack Puller

haystack_puller.py pulls points, equipment, and history from a remote Project Haystack REST server — Haxall, SkySpark, or any other Haystack-compliant server — and ingests it through the exact same pipeline every other source uses (ingest.py, mapping.py). Point it at a read-only API user on an existing Haxall/SkySpark deployment and Baler classifies the same already-tagged points/equipment through its own Brick mapping, for comparison, with no change to the target system.

This is the opposite direction from Haxall Drop-In: that page installs a Fantom extension into Haxall so it pushes history to Baler. This is pull-mode — Baler reaches out to a remote Haystack server on its own poll loop, nothing is installed on the other side.

Enabling it: Ops → Sources (recommended)

Sign in as an admin, open Ops → Sources → Add source, pick Haystack server, and fill in the URL, user and password. Enable it and the always-on source_runner service starts pulling within 30 seconds, on the interval you set. The Sources table shows each source's health: ok, failing with the error from its last run, misconfigured with the settings that are missing, or pending / disabled. Passwords are encrypted at rest and never shown again. The same page configures SQL, Open-Meteo and ENERGY STAR sources. API: GET/POST /ops/sources, PATCH/DELETE /ops/sources/{id} (admin).

Enabling it from env (older installs, one-off imports)

It's not part of the default stack — plain docker compose up -d does not start it. It's gated behind a compose profile:

Code
docker compose --profile haystack-pull up -d haystack_puller

Set REMOTE_HAYSTACK_URL and REMOTE_HAYSTACK_USER first. The container doesn't actually enforce this: docker-compose.yml passes both through as ${VAR:-}, so they're always present in the container's environment (blank if you never set them), and the code's "required" check only fires when the variable is completely absent — which it never is under docker compose. Leave them unset and the container starts clean, then fails every pull cycle against an empty URL and retries silently forever, with no startup error to tell you why. Only running the module directly (python -m baler.haystack_puller, no env var set at all and no --haystack-url flag) actually hits the "required" argparse error. See the repo's .env.example or the table below.

How it works

• Auth: haystack_client.HaystackClient implements Haystack's SCRAM-SHA-256 handshake from scratch over requests — no third-party Haystack client dependency.
• Progress tracking: haystack_pull_state.py keeps a small Postgres watermark table (haystack_pull_checkpoint, keyed by point URI) so a restart resumes from the last successful pull instead of re-fetching history from scratch.
• Ingest: routes through ingest_haystack_equip_tags, ingest_tags, link_equip_ref, and ingest_reading — the same functions mqtt_listener.py and ingest_api.py use — so results land in Oxigraph and Postgres identically to any other source, and equipment gets classified via rules/haystack_equip_to_brick.yaml.
• Each remote ref is namespaced as <source_id>:<ref> so multiple sources can't collide.

No real Haxall/SkySpark instance handy?

scripts/mock_haystack_server.py runs a small fake Haystack server in-process (real SCRAM-SHA-256 handshake, canned equip/point/history grids seeded with an AHU, a VAV, and a couple of already-tagged points — one deliberately untaggable, so you see a miss in the classification summary too) — enough to try the puller end-to-end without any external system:

Code
python -m scripts.mock_haystack_server
# prints: mock Haystack server on http://127.0.0.1:<port>
#         --haystack-user tduser --haystack-password tdpass

Then, against the printed port:

Code
python -m baler.haystack_puller \
  --haystack-url http://127.0.0.1:<port> \
  --haystack-user tduser --haystack-password tdpass \
  --once

Configuration

Env var	CLI flag	Default	Required
REMOTE_HAYSTACK_URL	--haystack-url	—	yes
REMOTE_HAYSTACK_USER	--haystack-user	—	yes
REMOTE_HAYSTACK_PASSWORD	--haystack-password	""	no
REMOTE_HAYSTACK_SOURCE_ID	--source-id	haxall	no
—	--point-filter	point	no
—	--equip-filter	equip	no
—	--tick-seconds	300.0	no
—	--backfill-days	7.0	no
—	--once	off	no

--point-filter/--equip-filter are Haystack filter expressions passed straight to the remote server's read op. --backfill-days is how far back history is pulled the first time a point is seen; after that, the watermark table takes over. --tick-seconds is deliberately coarse — it polls a remote server, not Baler's own MQTT broker.

To run it standalone, outside compose:

Code
python -m baler.haystack_puller \
  --haystack-url https://your-haxall-host \
  --haystack-user readonly

One-time migration

The same pipeline doubles as a Haystack/Xeto-to-Brick migration tool, not just a live shadow of an existing deployment. --once runs a single pull-everything pass — every currently-tagged equip and point on the source — and exits instead of polling forever:

Code
python -m baler.haystack_puller \
  --haystack-url https://your-haxall-host \
  --haystack-user readonly \
  --once

It prints a classification summary (direct/fallback/miss/llm counts for equip and points separately). Follow up with python -m baler.autotag to run the model step on anything the rule engine alone couldn't place; what it can't settle becomes a mapping proposal for a person to review — see Adding a Data Source and the README's autotag section.

This works unmodified against a Xeto-enabled Haxall/SkySpark server, too. Xeto adds a new, opt-in "version 5" wire protocol (Jeto encoding), but a server only serves it to a client that explicitly requests it via an Xeto-Version: 5 header — haystack_client.py never sends one, so it gets the same classic read/hisRead JSON either way. There's no separate "Xeto migration" path because there's nothing Xeto-specific to bridge at the wire level.

Known v1 limitations

• Poll-only — no Haystack watchSub/subscription support.
• Single remote source per process — run multiple haystack_puller instances (different --source-id) if you need to pull from more than one remote server.
• A single failed pull cycle (remote hiccup, transient auth failure) logs and retries on the next tick rather than crashing the process.

This is also the reference implementation for any future pull-style source (ENERGY STAR, a utility data API, ...) — see Adding a Data Source for what's generic here versus what's specific to Haystack.

Importing export files instead of pulling

When you have files rather than a live server (a SkySpark project you're leaving, a Haxall or nhaystack dump), haystack_import loads them through the same path the puller uses:

Code
python -m baler.haystack_import --source-id skyspark-hq records.zinc his-*.zinc

• Records: any grid of site/equip/point rows with ids, e.g. readAll(site or equip or point) saved with ioWriteZinc. They get the same identities and Brick classification as a live pull, so a later haystack_puller --source-id skyspark-hq against the same server updates the same points.
• History: hisRead grids (meta id, columns ts/val), written with their original timestamps. Record files are always imported first, whatever order you list them in.
• Zinc, Haystack 3 JSON, Hayson and Trio are all accepted (sniffed; or --format). Needs the haystack extra (openhay). Re-importing the same files is a no-op.