Look up the stages, reports and outputs of a single-region local run (also used on a cloud profile VM). Start one with the quickstart; diagnose failures with Troubleshooting.
A region is one of:
<council>/<town>/<suburb> from the LINZ
NZ Suburbs and Localities layer, e.g.
queenstown-lakes-district/arrowtown/arrowtown. Find one with
tools/find_area.py; see Choosing an area. No
config.py edit is needed. Its outlines and bounding box are written by
src/localities.py to data/regions/<id>/ (building_outlines.geojson,
locality.json) at preflight, selected from the council’s LINZ GeoPackage
(see Building outlines source).config.REGIONS, such as pilot
(the original central Queenstown pilot) or town_gorge_north. Deprecated
for new work: these still build, and the pipeline prints a warning.Related settings:
src/surveys.py: discovers covering LINZ sources from public layer
metadata and keeps its metadata cache under ignored data/sources/linz/.
The selected layer IDs are recorded in the estimate and pinned for the job.config.CLOUD_SHARED_BUILD["local_region"]: the local/profile default
region, a locality id. Shared-cloud mode defaults to
config.CLOUD_SHARED_BUILD["regions"]. Both sit in the block that build
keys leave out, so changing a default rebuilds nothing.
config.PIPELINE_REGION is the deprecated fallback when local_region
is absent. Pass a region id to override any default.Region ids are relative paths: a locality’s outputs are under
data/regions/<council>/<town>/<suburb>/ and data/out/<council>/<town>/<suburb>/.
Flat files named after a region (build markers, logs) replace / with __.
LINZ NZ Building Outlines
are the source of truth for which buildings a region holds. src/linz_exports.py
downloads them once per territorial authority through the LINZ Exports API
(GeoPackage, EPSG:2193, cropped to the council boundary from LINZ layer 3036)
and caches them:
data/sources/linz/layer-101290/<council>/building_outlines.gpkg
data/sources/linz/layer-101290/<council>/manifest.json
The manifest records the LINZ layer version, publish date, export id,
download time and feature count. The cached file is reused until LINZ
publishes a new layer version. A locality’s outlines are then selected
locally with the same rule the WFS filter used (suburb_locality,
town_city, territorial_authority); both sources give identical
buildings. To fetch or refresh a council by hand:
.venv/bin/python src/linz_exports.py fetch "Queenstown-Lakes District"
.venv/bin/python src/linz_exports.py fetch "Queenstown-Lakes District" --refresh
If the export cannot be used (no export permission on the key, LINZ queue
failure), the locality falls back to LINZ WFS and locality.json records the
reason under outline_source. Set SOLAR_OUTLINE_SOURCE=wfs to use WFS
directly. LINZ also offers GeoParquet exports; the pipeline does not use
them yet.
Complete Local setup for system tools, Python and credentials. Additional operating limits:
data/models/. Without them it is skipped.--mode cloud --profile NAME runs this sequence in the existing developer-VM
checkout; update that checkout first. Shared cloud mode instead uses the
district engine, not the table below. Cloud commands are in the
cloud quickstart; exact backend differences are
in the execution reference.
Local runs now submit a task to a loopback coordinator and start one worker
process by default. No Google account, GCS bucket, gcloud or SQLite is
needed for queue operation. Fetching source data still needs the LINZ
credentials and dependencies described in Local setup.
Process cleanup is tested locally on macOS; CI is configured to run these
tests on Linux. Use WSL for Windows, not native Windows processes.
bash quickstart.sh pilot --no-open-browser
bash quickstart.sh pilot --workers 2 --no-open-browser
Each invocation still builds one region, with the same numbered stages below. Extra workers do not split that region. Tasks in the same checkout serialize because they share caches, stage markers and the combined map. Use separate, human-managed checkouts and data directories for concurrent builds; do not symlink their writable data trees together. The coordinator does not create worktrees, allocate CPU/RAM or manage separate environments.
The printed queue directory defaults to data/pipeline_queues/<id>/.
--queue-dir PATH chooses a persistent directory instead. It must not be
owned by another running coordinator; it is not an option for attaching to
one. A directory can retain prior tasks, including failures; prefer a new
directory for a new run.
For debugging, start one coordinator in a terminal:
.venv/bin/python -m tools.pipeline_queue serve --queue-dir data/my-local-queue
In another terminal, submit tasks before starting workers:
.venv/bin/python -m tools.pipeline_queue submit --queue-dir data/my-local-queue \
pilot --no-open-browser
.venv/bin/python -m tools.pipeline_queue submit --queue-dir data/my-local-queue \
town_gorge_north --no-open-browser
Run each worker command in its own terminal, using distinct names:
.venv/bin/python -m tools.pipeline_queue worker --queue-dir data/my-local-queue --worker one
.venv/bin/python -m tools.pipeline_queue worker --queue-dir data/my-local-queue --worker two
These two tasks serialize in the same checkout. To exercise concurrent
builds, submit a task with --workspace /absolute/path/to/another/checkout
and --python /absolute/path/to/its/.venv/bin/python. That checkout needs
the queue implementation, its dependencies and its own prepared environment.
Workers exit when the submitted batch is terminal; start workers again
after submitting more tasks.
Inspect status while the coordinator runs:
.venv/bin/python -m tools.pipeline_queue status --queue-dir data/my-local-queue
state.json persists task specifications, worker/attempt ownership and
terminal outcomes. attempts/<task>-<attempt>.log retains subprocess output.
Existing data/pipeline_runs/ reports and preview URLs are unchanged.DEGRADED steps..connection.json holds a local
access token and is owner-readable; never share it or publish queue files.
This is a trusted local developer tool, not a multi-user/public service.data/pipeline-workspace.lock/ detects a conflicting queued build and
fails it visibly. Older/direct stage scripts do not honor that lock:
humans must coordinate those operations.state.json, attempt logs and lock owner.json
files. Stop any surviving workers/child processes before manually removing
the specific stale coordinator.lock/ or pipeline-workspace.lock/
directory. Restarting a coordinator marks previously running tasks
interrupted, never successful. Do not remove locks on a live run.The existing GCS fleet queue is not migrated by this first local slice. It still has different task, retry and regional-execution semantics; see the consolidation boundary.
Steps 04–17 are the shared stage plan (src/build_plan.py),
the same per-region steps the district and fleet engines run; steps 12 and
18–21 are local-only. The local pipeline writes stage markers but does not
pass --skip-done: rerunning creates a new report and reruns the commands,
with any input reuse handled by individual tools.
| # | Step |
|---|---|
| 01 | Preflight: validate area and runtime |
| 02 | Fetch outlines, elevation, imagery, point cloud; ensure the wide DEM (src/fetch_regions.py, src/fetch_dem_wide.py) |
| 03 | Vision precompute (optional) |
| 04 | Fit outlines to the roof (optional; SOLAR_OUTLINE_FIT=0 skips) |
| 05 | Agree identical roofs (optional; SOLAR_REPEAT_ROOFS=0 skips) |
| 06 | Register imagery alignment (optional) |
| 07 | Build roof geometry and layout |
| 08 | Gate panel layouts |
| 09 | Screen demolished buildings (paid referee only if approved, below) |
| 10 | Rerank panel layouts |
| 11 | Derive building solar potential |
| 12 | Render building verification report |
| 13 | Patch roof confidence |
| 14 | Bake per-building horizons |
| 15 | Build per-pixel solar heat map |
| 16 | Add building addresses (optional) |
| 17 | Emit regional map tiles (src/emit_region.py) |
| 18 | Combine map dataset (src/combine_regions.py) |
| 19 | Build local 3D terrain tiles |
| 20 | Validate map contract |
| 21 | Start local map preview |
Paid step question. Before queueing (local) or inside the cost
confirmation (cloud), the pipeline asks whether to use the paid demolition
referee, with an estimate. Only yes approves; Enter, a non-interactive
run or --yes declines. --paid-referee / --no-paid-referee answer it in advance.
See demolition screening.
Unattended and interactive runs. --yes (-y) accepts the cloud cost
estimate without a prompt; the estimate is still printed and logged. Review
gates never block a run by default: a failed deploy gate on the shared cloud
preview prints GATE WARNING and continues. --interactive stops and asks
at a failed gate instead (it cannot be combined with --yes). The local
pipeline has no review gate, so --interactive changes nothing there. See
cloud quickstart: deploy gate.
Exact definitions: tools/pipeline.py.
data/pipeline_runs/<UTC-run-id>/: report.md (status, timing and effect
of each step), status.json (the same step statuses and timings as JSON,
without commands; read by the web pipeline), run.log
(pipeline start and finish timestamps plus step output tagged [QS-NN] and
run timing tagged [RUN]), and the preview server’s PID and log. Step
completion and total runtime are shown as hh:mm:ss; the start and finish
timestamps are UTC. A local run prints PIPELINE_RUN_DIR=<path> when it
creates this folder.
--no-preview-server passes step 21 without starting a server, for callers
that already serve the map.data/out/<region>/: the region’s emitted tiles and summaries.data/: the combined map. Step 18 replaces whatever map data was there
with the selected region. Other emitted regions remain under data/out/
but are not included in this combine. Don’t run it over map data you want
to keep. Root data/solar_potential.geojson is not a current map input.data/regions/<region>/quickstart_report.html: per-building check cards.The preview serves the map and combined data on 127.0.0.1 and needs
internet access for basemaps. It does not expose the pilot-only refit API
(execution boundaries).
The pipeline starts the preview server in a separate session; it keeps running
after the pipeline finishes, and Ctrl+C in the pipeline terminal does not
stop it. Inspect the run’s recorded PID and verify it is still the preview
server before stopping it:
cat data/pipeline_runs/<run-id>/map-preview-server.pid
ps -p 12345 -o pid=,command=
kill 12345
Replace 12345 with the PID from the first command. Confirm the command names
tools/pipeline_serve.py from this checkout before sending the signal. Plain
kill sends SIGTERM, which stops the preview even if it inherited an ignored
SIGINT from a background launch. This stops only the preview HTTP server; it
does not cancel a pipeline build or delete generated data. If you started
tools/pipeline_serve.py directly in a terminal, Ctrl+C there also stops it.
If the PID file is missing
or stale, find the listener on the port shown in the preview URL with
lsof -nP -iTCP:<port> -sTCP:LISTEN, inspect its command with ps, and stop
it only if it is the preview server. A browser tab whose server is no longer
listening is simply stale; close the tab rather than killing an unrelated
process.
PASS: ran and its output checks passed.DEGRADED: carried on without a source or capability. Read the effect.SKIPPED: optional step not run (usually vision).FAIL: stopped. Later steps show NOT RUN.A finished run can still be degraded. Missing LiDAR weakens the geometry and can leave the heat map empty; missing imagery, addresses or terrain each lose their part. Read the report before trusting a result.
Do not delete data/ wholesale. It contains curated labels, benchmarks,
verdicts and trained models as well as generated files. Do not delete a
whole region directory without inspecting it either: it can contain review
results and provenance you need to keep. Preserve .env and .venv.
To reclaim local space, identify regenerable inputs for the selected region before removing them. The wide DEM and point-cloud tiles can be shared by neighbouring regions. The regional publication cleanup checks upload sizes and shared-tile ownership; it is not a blanket cleanup command for a local pipeline run.
git clean -fd leaves ignored inputs but deletes untracked work, including
new labels or documentation. Preview with git clean -nd and inspect every
path before using it. Do not use git clean -fdX to reset this project.