nz-solar-potential

Dataset operations

Prepare, build, review and release a district dataset. Run commands from the project folder with .venv active. Prerequisites: local setup, team access.

District builds run on the build VM, never on a laptop. A laptop builds one small region at a time (see the quickstart). For a shared-VM developer preview rather than a release, use the cloud quickstart.

The flow

flowchart TB
  Config["Choose regions in config.py"] --> Fetch["Fetch missing regional inputs and wide DEM"]
  Fetch --> Ownership["region_build.py: prepare outline ownership"]
  Ownership --> Predict["Optional selected-face precompute: predict_faces.py"]
  Predict --> District["run_district_build.sh on build VM"]
  District --> Served[("Per-region outputs + combined served set")]
  Served --> Review["Inspect outputs and run predeploy_check.py"]
  Review --> Gate{"Gate passes or production override reviewed?"}
  Gate -->|no| Fix["Correct inputs or code before rebuilding"]
  Fix --> District
  Gate -->|yes| Publish["publish_served.py: versioned tiles bucket"]
  Publish --> Switch["Update site-config.js release pointer"]

Arrows show the operational order, not that every command invokes the next. Input acquisition, ownership and face prediction are preparation outside the district script. The remote launcher can fetch with --fetch, but does not run face prediction. For stage order and differences between launchers, use the execution reference.

Source data

Regional DSM, DEM, LiDAR-index and aerial-photo layers are discovered from LINZ metadata by src/surveys.py and cached locally under ignored data/sources/linz/. Estimates pin the selected layer IDs for the job.

Data Source Used for
Building outlines, layer 101290 LINZ Data Service (WFS 1.0.0) The buildings estimated. src/fetch_data.py, src/fetch_regions.py
Regional LiDAR 1 m DSM and bare-earth DEM LINZ Exports API; covering layers are selected from LINZ metadata Roof and obstacle heights, shading and heat map
LiDAR tile index and raw point cloud LINZ metadata and a verified public bulk source when available Point-level roof evidence; otherwise the build uses its DSM without stale tile manifests
Regional aerial imagery and survey-year reference imagery LINZ metadata; coverage and year are checked against the selected area and DSM Roof lines, obstructions and photo alignment. Imagery is optional
NZ 8 m DEM, layer 51768 LINZ Distant horizons. Fetched to data/dem_wide_mosaic.tif by src/fetch_dem_wide.py
NZ Addresses, layer 123113 LINZ Address labels and search. src/add_addresses.py
Measured sunlight NIWA SolarView GHI (Wakatipu basin); else the nearest NIWA measured-radiation station Sunlight calibration. src/solar_model.py
ERA5 daily radiation, 1940–2025 Copernicus, via Open-Meteo (CC BY 4.0) Scales calibration between places; low-solar runs. climate/era5/, tools/pack_era5_daily.py
District plan zones Queenstown Lakes District Council Home or business pricing. tools/fetch_zoning.py
Aerial basemap LINZ Basemaps Map display; reference photo for lean. rooftop-solar.html, src/register_imagery.py
World imagery basemap Esri World Imagery Low-zoom map display only

Where there is no LiDAR (Cromwell, Glenorchy), the survey is marked "elevation": "synthetic" and the build uses a modelled surface (src/synthesize_dsm.py).

1. Choose an area

Check LINZ has outlines, DSM and imagery for it, then add the region and its bounding box to config.REGIONS. The original pilot is pilot; its inputs start in data/ and are linked into data/regions/pilot/.

2. Fetch

.venv/bin/python src/fetch_regions.py frankton_flats   # one region
.venv/bin/python src/fetch_regions.py                  # every region
.venv/bin/python src/fetch_dem_wide.py                 # wide DEM, if missing or too small

Each region lands in data/regions/<region>/. Fetching resumes: existing files are skipped. Imagery is fetched in chunks of at most 8 km². Areas with no imagery log a warning and build from LiDAR alone. src/fetch_data.py fetches the legacy pilot inputs into data/.

3. Prepare

After fetching every region you will combine, give each building one owning region:

.venv/bin/python src/region_build.py

4. Build

Prepare selected-face readings with tools/predict_faces.py --region REGION when using the selected-faces geometry path. The district script consumes existing readings; it does not run that precompute itself. Before a geometry build, authorize demolition screening, including external API cost and image transfer. Then, on the VM:

bash src/run_district_build.sh                       # incremental: stale buildings only
bash src/run_district_build.sh --regions "pilot frankton_flats"
bash src/run_district_build.sh --force               # rebuild everything
bash src/run_district_build.sh --yield-only          # solar model changed: recompute kWh, no LiDAR

Check data/build_logs/<region>.log and pre-plan warnings. Outline fitting and repeated-roof consensus can fail without stopping the build; they run before normal/forced builds, not yield-only builds. A failed region stops the final combine.

--regions restricts the build loop, not the final combined map. Combine includes available summaries under data/out/, excluding config.SERVED_EXCLUDE; unchanged regions can be included.

The execution reference explains stage order, sidecars and marker freshness. Builds are incremental from per-building keys; local pipeline and worker reruns have different reuse rules.

For fast layout work on one area: bash src/run_dev_loop.sh pilot.

Demolition screening

Every engine runs the demolished stage. Its free screen runs every time; its referee, which sends survey/current image crops to the Anthropic API, is off unless approved at the start of the run. Each entry point asks a yes/no question with a conservative estimate (USD $0.004 a roof, stopping near SOLAR_DEMOLISHED_MAX_USD, default USD $5 a region; not a spending cap) and approves only on an explicit yes. Enter, any other answer and non-interactive runs decline. The quickstart’s VM/preview estimate does not include this cost.

Non-interactive approval: --paid-referee on tools/pipeline.py and tools/remote_build.sh, the paid_referee input on the GitHub workflow, or SOLAR_DEMOLISHED_REFEREE=1 in the environment of tools/fleet.sh up or a direct engine call. --no-paid-referee declines without asking, as does --yes on tools/pipeline.py (it approves the VM cost estimate only). The execution reference lists where each entry point asks.

SOLAR_DEMOLISHED=0 disables the stage; with the referee declined, cached verdicts still apply. Yield-only builds skip screening.

Missing imagery/key, API failures or exhausted usage can leave only cached verdicts applied; a zero exit code does not prove every roof was screened. The stage implementation defines the cache, hand-drawn exemptions and withholding criteria.

5. Check

  1. Read each region’s log for failures and warnings.
  2. src/audit_layout_quality.py: low fill, scattered arrays, mixed angles, too much obstruction.
  3. src/render_top_movers.py: picture cards of the buildings that changed most.
  4. src/render_building_debug.py <building_id>: one roof’s faces, planes and panels.
  5. src/validate_obstructions.py when obstruction rules change.
  6. For layout changes, see “Judging a layout change” in AGENTS.md (tools/layout_ab.py, tools/layout_health.py, tools/face_area_check.py).
  7. Look at the map: a mix of roofs, known hard cases, areas without imagery.

When code changed, run the matching tests before pushing; the reviewer’s guide lists the fast suite, synthetic build and geometry checks.

6. Serve locally

.venv/bin/python tools/pipeline_serve.py --root . --port 8000

Open http://localhost:8000/rooftop-solar.html. This loopback server supports the byte-range requests PMTiles needs. Stop it with Ctrl+C.

The live refit server is a separate developer tool requiring root-level pilot inputs, not the regional preview path; see execution boundaries.

7. Publish

The site (rooftop-solar.html, the Build data page and their scripts) and rendered documentation are deployed by the static Pages workflow. The data is served from the Google Cloud tiles bucket, one folder per release: gs://solar-potential-map-tiles/v<N>/. site-config.js (dataBase, dataVersion) tells the page which folder to read.

From GitHub (Actions > “Build and publish the map”) or a laptop (tools/remote_build.sh --publish), the VM builds, gates and publishes. To review a VM build on a laptop first:

tools/deploy_from_vm.sh                          # pull the served set, run the gate, stop
tools/deploy_from_vm.sh --push                   # ...and publish if the gate passes
tools/deploy_from_vm.sh --push-reviewed "why"    # publish past a gate whose list you have checked

The gate is tools/predeploy_check.py: it compares the new build against the live one. Publishing (tools/publish_served.py) uploads a new version folder, keeps the newest three, and commits only the site-config.js change. A version folder is never changed after upload.

Roll back by pointing site-config.js dataBase and dataVersion at an older folder that is still in the bucket.

Safety