nz-solar-potential

Build data

The local Build data workspace is one list of areas beside a map. Tick areas, build them, follow the queue and stage the results without leaving the list: each row’s status updates in place. Advanced, rarely used settings (the national area catalogue, local staging, local releases and retention) are on the ⚙ Config page. A target toggle at the top picks where the work applies:

Target Shows A build runs Costs
Local Prepared and built areas in this checkout, plus its cached datasets tools/pipeline.py <region> on this computer, as the quickstart does Nothing in the cloud
Developer Built areas in developer previews on the shared VM tools/pipeline.py "<regions>" --mode cloud --cloud build on the shared build VM, publishing to your own devel/<alias>/ preview, as the cloud quickstart does Google Cloud, shown before you start
Published Built areas in the live map data version tools/remote_build.sh --regions "<regions>" --publish on the shared build VM; it publishes only if the deploy gate passes Google Cloud, shown before you start

The panel never publishes past a failed gate: --reviewed is left to the command line, after someone has rendered and judged the gate’s list.

Start it

Do the local setup first; the panel needs the same Python environment and LINZ API key. The Developer and Published targets also need gcloud signed in with access to the shared project (team access) and the GitHub CLI for your preview alias.

.venv/bin/python tools/pipeline_web.py

It prints a link such as http://127.0.0.1:8766/pipeline.html#token=… and opens it. Options:

Option Meaning
--port N Listen on another loopback port (default 8766; the map preview uses 8765).
--python PATH Interpreter for pipeline runs (default: the one running the panel).
--no-open-browser Print the link without opening it.
--staging-root PATH Store staged copies here instead of ignored data/regional_staging/. Use an untracked location outside emitted outputs.
--release-root PATH Store isolated local candidates, deployment state and purge records here instead of ignored data/local_releases/. Must not overlap staging or working outputs.

Ctrl+C stops the panel. Running builds are cancelled first and queued ones are dropped. Cancelling a cloud build stops the local command; the VM is stopped only if that build started it.

Closing the browser tab does not stop the panel. If its terminal is still open, return to it and press Ctrl+C. If the terminal is gone, starting a new panel on the same port fails and names the old one:

Cannot listen on 127.0.0.1:8766 (Address already in use).
An earlier pipeline panel is still running (process 12345).
Stop it with Ctrl+C in its terminal, or from here with:
  kill 12345

Run the kill command it prints. The panel treats kill (SIGTERM) and Ctrl+C (SIGINT) the same way: it cancels running builds and drops queued ones before exiting, including when it was started in the background. If the port belongs to another program, the message shows that program’s command instead; check it yourself with:

lsof -nP -iTCP:8766 -sTCP:LISTEN
ps -p 12345 -o pid=,command=

For a panel started with --port N, use N instead of 8766. Check that the listener is gone by repeating the lsof command. Do not stop a process just because it owns the port if its command is unrelated.

Use it

Areas

The list is a tree of LINZ building-outline levels: council (territorial authority) › town or city › suburb or locality. Only areas with buildings appear; water such as lakes and inland bays is left out. The tree starts collapsed to councils. Each row is one line: tick box, name, building count and a status dot; click ▸ to expand a row and see its options, its last completed build and every earlier attempt (latest first).

Status colours are shared by the list dots, the map fill and the legend: Not built, Queued, Running, Built, Staged, Live and Failed. Click a legend entry to hide or show areas in that state. Status is per target: Local reads this checkout’s runs and staging; Developer and Published read the cloud data catalogue. Local areas that are not in the national catalogue (for example pilot) are listed under Other local areas.

The sticky bar under the list sums the selection (areas, buildings and a rough compute time from this checkout’s measured rate; downloads are extra and appear in the Build… estimate) and offers the actions the selection allows. Click the selection count to list the selected areas and untick any of them; the list stays open until you close it.

Build

The build dialog estimates every selected area and applies one set of options to all of them. Per area it shows:

Aerial imagery is included by default and can be unchecked. That skips its download, alignment, image-based obstruction checks, and paid AI referee; outlines, elevation and LiDAR remain unchanged. The estimate recalculates when the imagery or paid-referee choice changes, and the resulting choices are pinned to the job. The paid AI referee is off by default and is only available when imagery is included.

Covering LINZ elevation and imagery layers are discovered from public layer metadata, cached under ignored data/sources/linz/, and pinned from estimate through execution. Areas with no covering LiDAR DSM use the warned synthetic surface path; they are not blocked by a static survey list.

For Developer and Published the estimate also shows the Google Cloud cost (see Cloud builds). Cloud totals sum the separate jobs’ VM/startup/storage allowances rather than assuming a single combined run. The queue button is enabled after checking the cost-acknowledgment box. Local builds queue directly from Add to queue. Paid options, such as the AI referee, start unticked every time the dialog opens, whatever was chosen before; the estimate shows their upper cost when ticked.

Grid cells

Grid cells are a build option, not a list level. Use them for a quick developer check of a small area, or for a locality too large to build on one computer in reasonable time. In the build dialog (Local only), Split each locality into grid cells offers Off (default), Auto or a fixed 2×2 to 10×10 grid. Auto splits only localities over 12,000 buildings, into N×N cells with N = ceil(√(buildings / 12,000)), clamped to 2–10, so each cell holds roughly 12,000 buildings or fewer. Cells with no buildings are dropped. The cells then appear as a fourth level under their locality, and stay there until the locality is built whole.

When every estimated cell has built, the locality row offers Combine. That queues an ordinary whole-locality build. It reuses the downloaded sources but reruns geometry and yields. Once it succeeds, the cells are hidden again. Cells are never staged or published.

Each locality boundary is the LINZ suburb/locality polygon, not a council boundary. The grid divides its bounding box and clips each cell to the polygon. It needs no additional API key or dataset.

The estimate reports the selected building count, the sum of disjoint buffered acquisition footprints, download estimate, runtime, disk estimate and warnings. These are planning estimates, not limits. A partial-area build runs locally only; it cannot start a Developer or Published build. Each selected cell is estimated and built as its own work area and queue job. The estimate lists each cell separately and totals their costs/resources; shared source caches may reduce actual downloads on later jobs.

Buildings are selected as whole LINZ building-outline features. For each outline, the pipeline finds its representative point in EPSG:2193 and assigns it to the lexically lowest selected-area ID whose polygon covers that point. Only buildings owned by a selected grid cell are included; boundaries never clip buildings and ties cannot duplicate them. This is a stable rule for an identical parent, grid size and selection.

Each selected cell has a deterministic selection ID below its parent: data/regions/<parent>/grid-<16-hex-id>/. Its selection.json records the parent, selected IDs, boundary dataset/version and geometry, ownership rule, building IDs and acquisition footprints. Pipeline artefacts are written below that selection directory, and emitted artefacts below the matching data/out/ directory; the parent locality’s files are not changed. Each run initially combines its own cell into the local map data. The last job of a multi-area local submission combines existing emitted regions/cells once, without rerunning geometry or yields. Cancelling that final job also cancels its combine. Partial outputs remain partial, not complete parent coverage. Selections are excluded from normal complete-area listings and are never published as complete coverage.

Queue and jobs

The Queue and Jobs panels sit over the map.

Queue cards refresh every five seconds and use the same progress calculation as job details. Completed and deliberately skipped stages advance progress, weighted by historical stage durations when available; interrupted, failed and unrun stages do not count as complete. Percentages can remain still during a long stage and are approximate, not elapsed-time animations. Only a successful job reaches 100%, after final combining or publishing. Preparation and cloud builds without measured stage telemetry show an indeterminate indicator instead of an invented percentage. Cloud stage details are not currently streamed from the VM.

An exhausted runtime estimate reads Taking longer than estimated, not zero seconds remaining. Last activity measures log/status writes, not a worker heartbeat; a quiet stage is not necessarily stuck. If updates fail, the last known cards remain visible with a disconnected warning; progress bars are hidden until updates reconnect so stale work does not appear live.

Estimates are planning assumptions, not limits; actual time and download can be higher. The time per building comes from the median of this checkout’s last ten finished runs, or the defaults in estimate_defaults.json, which record where each figure was measured. An estimate expires after 30 minutes; a build can only start from the estimate that was shown.

Config page

⚙ Config holds settings that are rarely needed.

What a build does

Map layers

Layers & legend, below the navigation controls, reuses the shared component and layer styling used by the canonical map. It groups analysis, derived and source datasets above the basemap. Panels paint above solar-analysis building outlines, which paint above the heat map. Basemap selection switches between the existing Esri/LINZ aerial combination and OpenStreetMap without resetting overlays.

For Local, the dataset catalogue checks actual files in the checkout, including shared cached sources and available intermediate outputs. A cached extent is drawn as a faint fill with a dashed boundary; it is not a claim that every pixel contains usable data. Missing requested LiDAR tiles are not shown as cached. Layer data is loaded on demand when enabled.

Prepared/built region and locality-selection boundaries are separate Build areas overlays. Their status colours are not evidence that all sources were downloaded. Refresh, area-status refreshes and completed jobs refresh the catalogue; existing dataset visibility is retained where the dataset still exists.

Developer and Published never inherit this computer’s cached-source overlays. Until remote source catalogues are supplied, the legend explicitly reports that those dataset layers are unavailable; their existing area listings and preview links remain usable.

Jobs run in two lanes, each one job at a time: the local lane on this computer and the cloud lane on the shared VM. A job started while its lane is busy waits in the queue. The lanes are independent, so a local build and a cloud build can run together.

Local builds:

Cloud builds

Each area’s run directory is data/pipeline_runs/<run-id>/, the same as a command-line run. It holds report.md, run.log and status.json (see Pipeline reference). The panel keeps its own records under data/pipeline_web/, which is ignored by Git:

The run directory also holds acquisition_timings.json. Each dataset record has its source/layer, UTC start, outcome, numeric elapsed seconds, and elapsed hh:mm:ss; transfer seconds and hh:mm:ss are separate where measurable. Each record is one dataset acquisition attempt, so a failed attempt and a later retry remain separately visible. Total acquisition time includes export queue/polling, transfer and local preparation; transfer time measures byte-transfer work only, not waiting or mosaicking. Where export polling or local preparation takes at least a minute, the log also gives a compact export_wait or local_prepare phase. Multipart exports and LiDAR tiles are summarized per dataset. The shared 8 m terrain DEM keeps disjoint selection footprints, each with the normal 30 km terrain buffer, rather than downloading the enclosing rectangle. Unavailable and failed acquisitions are recorded too; failure records contain the exception type, not a potentially sensitive URL or error message. Grid outline preparation happens during Estimate, before the pipeline run, and its timing is stored in data/regions/<parent>/grid-<16-hex-id>/acquisition_timings.json.

Security

The panel is not a hosted service; it runs on your computer with your gcloud credentials.

Troubleshooting

Symptom Fix
“Read-only: the panel server is not running” Start tools/pipeline_web.py and open the link it prints.
“Cannot listen on 127.0.0.1:8766 (Address already in use)” If the message names an earlier pipeline panel, run the kill <pid> it prints (Start it). If another program holds the port, start the panel with --port N or --port 0.
“This link’s token is out of date” The panel was restarted; open the new link.
“No LINZ API key found” Set LINZ_API_KEY as in local setup.
“Could not load grid” Check the LINZ locality and API key, then retry.
“No LINZ building outlines were found in the selected grid cells” Some grid cells contain no mapped buildings; select cells that cover buildings.
“zoom in” when choosing areas The view is wider than 1°; zoom in.
“a CLI build is running in this checkout” Wait for it, or stop it and see Local task queue before removing a stale lock.
An area failed Open its report from the Jobs section and follow Troubleshooting.
macOS “Python quit unexpectedly” after Estimate The panel process loaded PROJ before starting a child process; on macOS, PROJ’s SQLite fork handler can crash that child. Stop an older panel and restart it on current code to clear the loaded library state. Area calculations and local layer-catalogue generation run in a separate process (tools/pipeline_webui/geo_worker.py).
VM status UNKNOWN The message shows gcloud’s error; usually gcloud auth login or missing access to the project (team access).
“cannot publish: … unpushed commit(s)” Push or reset those commits, then estimate again.
A cloud job stopped at the code sync The VM was busy with another build; wait for it, then start again.