nz-solar-potential

Pipeline reference

Look up the stages, reports and outputs of a single-region local run (also used on a cloud profile VM). Start one with the quickstart; diagnose failures with Troubleshooting.

Regions

A region is one of:

Related settings:

Region ids are relative paths: a locality’s outputs are under data/regions/<council>/<town>/<suburb>/ and data/out/<council>/<town>/<suburb>/. Flat files named after a region (build markers, logs) replace / with __.

Building outlines source

LINZ NZ Building Outlines are the source of truth for which buildings a region holds. src/linz_exports.py downloads them once per territorial authority through the LINZ Exports API (GeoPackage, EPSG:2193, cropped to the council boundary from LINZ layer 3036) and caches them:

data/sources/linz/layer-101290/<council>/building_outlines.gpkg
data/sources/linz/layer-101290/<council>/manifest.json

The manifest records the LINZ layer version, publish date, export id, download time and feature count. The cached file is reused until LINZ publishes a new layer version. A locality’s outlines are then selected locally with the same rule the WFS filter used (suburb_locality, town_city, territorial_authority); both sources give identical buildings. To fetch or refresh a council by hand:

.venv/bin/python src/linz_exports.py fetch "Queenstown-Lakes District"
.venv/bin/python src/linz_exports.py fetch "Queenstown-Lakes District" --refresh

If the export cannot be used (no export permission on the key, LINZ queue failure), the locality falls back to LINZ WFS and locality.json records the reason under outline_source. Set SOLAR_OUTLINE_SOURCE=wfs to use WFS directly. LINZ also offers GeoParquet exports; the pipeline does not use them yet.

Needs

Complete Local setup for system tools, Python and credentials. Additional operating limits:

--mode cloud --profile NAME runs this sequence in the existing developer-VM checkout; update that checkout first. Shared cloud mode instead uses the district engine, not the table below. Cloud commands are in the cloud quickstart; exact backend differences are in the execution reference.

Local task queue

Local runs now submit a task to a loopback coordinator and start one worker process by default. No Google account, GCS bucket, gcloud or SQLite is needed for queue operation. Fetching source data still needs the LINZ credentials and dependencies described in Local setup. Process cleanup is tested locally on macOS; CI is configured to run these tests on Linux. Use WSL for Windows, not native Windows processes.

bash quickstart.sh pilot --no-open-browser
bash quickstart.sh pilot --workers 2 --no-open-browser

Each invocation still builds one region, with the same numbered stages below. Extra workers do not split that region. Tasks in the same checkout serialize because they share caches, stage markers and the combined map. Use separate, human-managed checkouts and data directories for concurrent builds; do not symlink their writable data trees together. The coordinator does not create worktrees, allocate CPU/RAM or manage separate environments.

The printed queue directory defaults to data/pipeline_queues/<id>/. --queue-dir PATH chooses a persistent directory instead. It must not be owned by another running coordinator; it is not an option for attaching to one. A directory can retain prior tasks, including failures; prefer a new directory for a new run.

Manually submit and run multiple workers

For debugging, start one coordinator in a terminal:

.venv/bin/python -m tools.pipeline_queue serve --queue-dir data/my-local-queue

In another terminal, submit tasks before starting workers:

.venv/bin/python -m tools.pipeline_queue submit --queue-dir data/my-local-queue \
  pilot --no-open-browser
.venv/bin/python -m tools.pipeline_queue submit --queue-dir data/my-local-queue \
  town_gorge_north --no-open-browser

Run each worker command in its own terminal, using distinct names:

.venv/bin/python -m tools.pipeline_queue worker --queue-dir data/my-local-queue --worker one
.venv/bin/python -m tools.pipeline_queue worker --queue-dir data/my-local-queue --worker two

These two tasks serialize in the same checkout. To exercise concurrent builds, submit a task with --workspace /absolute/path/to/another/checkout and --python /absolute/path/to/its/.venv/bin/python. That checkout needs the queue implementation, its dependencies and its own prepared environment. Workers exit when the submitted batch is terminal; start workers again after submitting more tasks.

Inspect status while the coordinator runs:

.venv/bin/python -m tools.pipeline_queue status --queue-dir data/my-local-queue

Evidence, conflicts and recovery

The existing GCS fleet queue is not migrated by this first local slice. It still has different task, retry and regional-execution semantics; see the consolidation boundary.

Local and profile-VM steps

Steps 04–17 are the shared stage plan (src/build_plan.py), the same per-region steps the district and fleet engines run; steps 12 and 18–21 are local-only. The local pipeline writes stage markers but does not pass --skip-done: rerunning creates a new report and reruns the commands, with any input reuse handled by individual tools.

# Step
01 Preflight: validate area and runtime
02 Fetch outlines, elevation, imagery, point cloud; ensure the wide DEM (src/fetch_regions.py, src/fetch_dem_wide.py)
03 Vision precompute (optional)
04 Fit outlines to the roof (optional; SOLAR_OUTLINE_FIT=0 skips)
05 Agree identical roofs (optional; SOLAR_REPEAT_ROOFS=0 skips)
06 Register imagery alignment (optional)
07 Build roof geometry and layout
08 Gate panel layouts
09 Screen demolished buildings (paid referee only if approved, below)
10 Rerank panel layouts
11 Derive building solar potential
12 Render building verification report
13 Patch roof confidence
14 Bake per-building horizons
15 Build per-pixel solar heat map
16 Add building addresses (optional)
17 Emit regional map tiles (src/emit_region.py)
18 Combine map dataset (src/combine_regions.py)
19 Build local 3D terrain tiles
20 Validate map contract
21 Start local map preview

Paid step question. Before queueing (local) or inside the cost confirmation (cloud), the pipeline asks whether to use the paid demolition referee, with an estimate. Only yes approves; Enter, a non-interactive run or --yes declines. --paid-referee / --no-paid-referee answer it in advance. See demolition screening.

Unattended and interactive runs. --yes (-y) accepts the cloud cost estimate without a prompt; the estimate is still printed and logged. Review gates never block a run by default: a failed deploy gate on the shared cloud preview prints GATE WARNING and continues. --interactive stops and asks at a failed gate instead (it cannot be combined with --yes). The local pipeline has no review gate, so --interactive changes nothing there. See cloud quickstart: deploy gate.

Exact definitions: tools/pipeline.py.

Outputs

The preview serves the map and combined data on 127.0.0.1 and needs internet access for basemaps. It does not expose the pilot-only refit API (execution boundaries). The pipeline starts the preview server in a separate session; it keeps running after the pipeline finishes, and Ctrl+C in the pipeline terminal does not stop it. Inspect the run’s recorded PID and verify it is still the preview server before stopping it:

cat data/pipeline_runs/<run-id>/map-preview-server.pid
ps -p 12345 -o pid=,command=
kill 12345

Replace 12345 with the PID from the first command. Confirm the command names tools/pipeline_serve.py from this checkout before sending the signal. Plain kill sends SIGTERM, which stops the preview even if it inherited an ignored SIGINT from a background launch. This stops only the preview HTTP server; it does not cancel a pipeline build or delete generated data. If you started tools/pipeline_serve.py directly in a terminal, Ctrl+C there also stops it.

If the PID file is missing or stale, find the listener on the port shown in the preview URL with lsof -nP -iTCP:<port> -sTCP:LISTEN, inspect its command with ps, and stop it only if it is the preview server. A browser tab whose server is no longer listening is simply stale; close the tab rather than killing an unrelated process.

Statuses

A finished run can still be degraded. Missing LiDAR weakens the geometry and can leave the heat map empty; missing imagery, addresses or terrain each lose their part. Read the report before trusting a result.

Cleaning up

Do not delete data/ wholesale. It contains curated labels, benchmarks, verdicts and trained models as well as generated files. Do not delete a whole region directory without inspecting it either: it can contain review results and provenance you need to keep. Preserve .env and .venv.

To reclaim local space, identify regenerable inputs for the selected region before removing them. The wide DEM and point-cloud tiles can be shared by neighbouring regions. The regional publication cleanup checks upload sizes and shared-tile ownership; it is not a blanket cleanup command for a local pipeline run.

git clean -fd leaves ignored inputs but deletes untracked work, including new labels or documentation. Preview with git clean -nd and inspect every path before using it. Do not use git clean -fdX to reset this project.