# Haystack Puller

`haystack_puller.py` pulls points, equipment, and history from a remote
[Project Haystack](https://project-haystack.org/) REST server — Haxall,
SkySpark, or any other Haystack-compliant server — and ingests it through
the exact same pipeline every other source uses (`ingest.py`, `mapping.py`).
Point it at a read-only API user on an existing Haxall/SkySpark deployment
and Baler classifies the same already-tagged points/equipment
through its own Brick mapping, for comparison, with no change to the
target system.

This is the opposite direction from [Haxall Drop-In](/haxall-drop-in):
that page installs a Fantom extension *into* Haxall so it pushes history
to Baler. This is pull-mode — Baler reaches out to a remote
Haystack server on its own poll loop, nothing is installed on the other
side.

## Enabling it: Ops → Sources (recommended)

Sign in as an admin, open **Ops → Sources → Add source**, pick *Haystack server*, and fill in the
URL, user and password. Enable it and the always-on `source_runner` service starts pulling
within 30 seconds, on the interval you set. The Sources table shows each source's health: `ok`,
`failing` with the error from its last run, `misconfigured` with the settings that are missing,
or `pending` / `disabled`. Passwords are encrypted at rest and never shown again. The same page
configures SQL, Open-Meteo and ENERGY STAR sources. API: `GET/POST /ops/sources`,
`PATCH/DELETE /ops/sources/{id}` (admin).

## Enabling it from env (older installs, one-off imports)

It's **not** part of the default stack — plain `docker compose up -d`
does not start it. It's gated behind a compose profile:

```bash
docker compose --profile haystack-pull up -d haystack_puller
```

Set `REMOTE_HAYSTACK_URL` and `REMOTE_HAYSTACK_USER` first. **The
container doesn't actually enforce this**: `docker-compose.yml` passes
both through as `${VAR:-}`, so they're always present in the container's
environment (blank if you never set them), and the code's "required"
check only fires when the variable is completely absent — which it never
is under `docker compose`. Leave them unset and the container starts
clean, then fails every pull cycle against an empty URL and retries
silently forever, with no startup error to tell you why. Only running the
module directly (`python -m baler.haystack_puller`, no env var set
at all and no `--haystack-url` flag) actually hits the "required"
argparse error. See the repo's `.env.example` or the table below.

## How it works

- **Auth**: `haystack_client.HaystackClient` implements Haystack's
  SCRAM-SHA-256 handshake from scratch over `requests` — no third-party
  Haystack client dependency.
- **Progress tracking**: `haystack_pull_state.py` keeps a small Postgres
  watermark table (`haystack_pull_checkpoint`, keyed by point URI) so a
  restart resumes from the last successful pull instead of re-fetching
  history from scratch.
- **Ingest**: routes through `ingest_haystack_equip_tags`, `ingest_tags`,
  `link_equip_ref`, and `ingest_reading` — the same functions
  `mqtt_listener.py` and `ingest_api.py` use — so results land in Oxigraph
  and Postgres identically to any other source, and equipment gets
  classified via `rules/haystack_equip_to_brick.yaml`.
- Each remote ref is namespaced as `<source_id>:<ref>` so multiple sources
  can't collide.

## No real Haxall/SkySpark instance handy?

`scripts/mock_haystack_server.py` runs a small fake Haystack server
in-process (real SCRAM-SHA-256 handshake, canned equip/point/history
grids seeded with an AHU, a VAV, and a couple of already-tagged points —
one deliberately untaggable, so you see a `miss` in the classification
summary too) — enough to try the puller end-to-end without any external
system:

```bash
python -m scripts.mock_haystack_server
# prints: mock Haystack server on http://127.0.0.1:<port>
#         --haystack-user tduser --haystack-password tdpass
```

Then, against the printed port:

```bash
python -m baler.haystack_puller \
  --haystack-url http://127.0.0.1:<port> \
  --haystack-user tduser --haystack-password tdpass \
  --once
```

## Configuration

| Env var | CLI flag | Default | Required |
|---|---|---|---|
| `REMOTE_HAYSTACK_URL` | `--haystack-url` | — | yes |
| `REMOTE_HAYSTACK_USER` | `--haystack-user` | — | yes |
| `REMOTE_HAYSTACK_PASSWORD` | `--haystack-password` | `""` | no |
| `REMOTE_HAYSTACK_SOURCE_ID` | `--source-id` | `haxall` | no |
| — | `--point-filter` | `point` | no |
| — | `--equip-filter` | `equip` | no |
| — | `--tick-seconds` | `300.0` | no |
| — | `--backfill-days` | `7.0` | no |
| — | `--once` | off | no |

`--point-filter`/`--equip-filter` are Haystack filter expressions passed
straight to the remote server's `read` op. `--backfill-days` is how far
back history is pulled the first time a point is seen; after that, the
watermark table takes over. `--tick-seconds` is deliberately coarse — it
polls a remote server, not Baler's own MQTT broker.

To run it standalone, outside compose:

```bash
python -m baler.haystack_puller \
  --haystack-url https://your-haxall-host \
  --haystack-user readonly
```

## One-time migration

The same pipeline doubles as a Haystack/Xeto-to-Brick migration tool, not
just a live shadow of an existing deployment. `--once` runs a single
pull-everything pass — every currently-tagged equip and point on the
source — and exits instead of polling forever:

```bash
python -m baler.haystack_puller \
  --haystack-url https://your-haxall-host \
  --haystack-user readonly \
  --once
```

It prints a classification summary (`direct`/`fallback`/`miss`/`llm` counts
for equip and points separately). Follow up with
`python -m baler.autotag` to run the [model step](/model-step) on
anything the rule engine alone couldn't place; what it can't settle
becomes a mapping proposal for a person to review — see [Adding a Data Source](/adding-a-data-source)
and the README's autotag section.

This works unmodified against a Xeto-enabled Haxall/SkySpark server, too.
Xeto adds a new, opt-in "version 5" wire protocol (Jeto encoding), but a
server only serves it to a client that explicitly requests it via an
`Xeto-Version: 5` header — `haystack_client.py` never sends one, so it gets
the same classic `read`/`hisRead` JSON either way. There's no separate
"Xeto migration" path because there's nothing Xeto-specific to bridge at
the wire level.

## Known v1 limitations

- **Poll-only** — no Haystack `watchSub`/subscription support.
- **Single remote source per process** — run multiple `haystack_puller`
  instances (different `--source-id`) if you need to pull from more than
  one remote server.
- A single failed pull cycle (remote hiccup, transient auth failure) logs
  and retries on the next tick rather than crashing the process.

This is also the reference implementation for any future pull-style
source (ENERGY STAR, a utility data API, ...) — see
[Adding a Data Source](/adding-a-data-source) for what's generic here
versus what's specific to Haystack.

## Importing export files instead of pulling

When you have files rather than a live server (a SkySpark project you're leaving, a Haxall or
nhaystack dump), `haystack_import` loads them through the same path the puller uses:

```bash
python -m baler.haystack_import --source-id skyspark-hq records.zinc his-*.zinc
```

- **Records**: any grid of `site`/`equip`/`point` rows with ids, e.g. `readAll(site or equip or point)`
  saved with `ioWriteZinc`. They get the same identities and Brick classification as a live pull,
  so a later `haystack_puller --source-id skyspark-hq` against the same server updates the same
  points.
- **History**: `hisRead` grids (meta `id`, columns `ts`/`val`), written with their original
  timestamps. Record files are always imported first, whatever order you list them in.
- Zinc, Haystack 3 JSON, Hayson and Trio are all accepted (sniffed; or `--format`). Needs the
  `haystack` extra (openhay). Re-importing the same files is a no-op.