National Siting Suitability Report

Spatial Compute Cost Optimization & Incremental ETL Guide

Executive Summary & Verified Batch Spend

During the development, benchmarking, and national scale-up of the AuraSiting Crafter AI data center suitability model, total cloud batch compute spend across dozens of full headless batch runs was ~$36 AUD (US$24.13).

• Verified Wherobots Cloud Spend: US$24.13 (Org ltq5l3obgb on aws-us-west-2).
• AUD Conversion: ~$36.01 AUD (at ~0.67 AUD/USD exchange rate).
• Batch Execution Efficiency: Incurred across ~35 automated batch pipeline runs, averaging ~US$0.69 (~$1.03 AUD) per full batch run.

Batch Run Breakdown & Compute Allocation

Cloud Cost Management & Runtime Lifecycle Best Practices

During initial developer setup, iterative query tuning, and interactive notebook experimentation, multi-session compute runs accumulated unexpected development costs against an early development invoice.

Following a joint resource utilization investigation with Wherobots engineering:

1. Resource Utilization Breakdown (80% Active / 20% Idle): Detailed platform analysis confirmed that 80% of incurred compute occurred while operations were actively executing during exploratory testing, with 20% attributable to idle resource utilization before auto-shutdown.
2. Built-In Platform Guardrails: Wherobots enforces built-in automated guardrails by default, shutting down inactive compute and notebooks with an 8-hour default Runtime TTL. Configurable idle timeout thresholds (15 / 45 / 120 minutes) are available directly in Runtime Settings.
3. Idle Timeouts as a Safety Net, Not a Strategy: As emphasized in the Wherobots Managing Costs Guide, automated idle timeouts provide a crucial safety net, but proactive shutdown remains the optimal development practice.
4. Mandatory Programmatic Teardowns: All Sedona and PySpark batch ETL scripts in this repository enforce strict try...finally: sedona.stop() and spark.stop() blocks to release compute instantly upon job completion.
5. High Headless Batch Efficiency: In contrast to interactive exploration, production headless batch runs across 15.91M national geometries consumed only US$24.13 (~$36 AUD) across ~35 full pipeline runs (~$1.03 AUD per run), proving the remarkable cost-efficiency of right-sized headless batch execution.

4 Core Cost Optimization Strategies

1. Decoupling Heavy Geometry Joins from Lightweight Multi-Criteria Scoring

Spatial siting pipelines consist of distinct computational tiers with vastly different resource requirements:

• Heavy Geometric Tier (Compute-Intensive): Ingesting raw vector feeds, reprojecting to GDA2020 (EPSG:7856), repairing invalid topologies (ST_MakeValid), constructing 30m riparian and 20m pipeline buffers, and computing ST_Difference developable overlays across millions of polygons.
• Lightweight Vector Scoring Tier (Compute-Light): Evaluating mathematical decay curves (S_power, S_sensitive, S_water) and weighted composite scores against precomputed distance attributes.

By structuring the pipeline as a directed acyclic graph (DAG) with intermediate materialized GeoParquet stages, tuning a scoring weight or modifying the sigmoidal acoustic threshold never triggers a re-run of the heavy geometric spatial joins. Only the downstream mathematical matrix recalculates.

2. Source-Level Data Fingerprinting & Snapshot Memoization

Authoritative baseline layers change infrequently (15.4M Geoscape cadastre parcels, 368k ABS meshblocks, 275k rail network vectors, 241k power grid features). Implementing cryptographic content hashing (ETags, GeoParquet file hashes, and Iceberg snapshot manifest IDs) ensures that untouched spatial tables are bypassed during batch execution, reading directly from cached Havasu storage partitions.

3. Delta Partition Processing (Apache Iceberg Time-Travel)

When state planning portals publish quarterly cadastral updates, leveraging Apache Iceberg's ACID snapshot metadata allows Sedona to isolate and process only modified parcel geometries (ST_Changes) rather than executing full continental scans.

4. Zero-Cost Client Compute Offloading

By compiling precomputed distance topologies into the standalone HTML report and offloading real-time multi-criteria exploration to in-browser JavaScript, millions of interactive public scenario evaluations occur at $0.00 cloud compute cost.

Cost Comparison: Full Scan vs. Incremental Pipeline

Reference & Engineering Playbook

For deep architectural patterns, Apache Sedona configuration flags, and memory management best practices, refer to the Wherobots & Antigravity Engineering Playbook.

Using cloud-optimized storage (Havasu/Iceberg tables) running on the Wherobots Cloud platform, we executed spatial queries over 16 authoritative national and state datasets:

Concrete Lakehouse Storage & Table Directory Structure

All spatial tables are cataloged under org_catalog.fgsdb.* on Wherobots Cloud and persisted directly in cloud object storage at s3://wherobots-cloud-us-west-2/org_ltq5l3obgb/fgsdb/ in AWS us-west-2 (GDA2020 / MGA Zone 56 projected CRS EPSG:7856 and GDA2020 geographic EPSG:7844).

Whitepapers, Engineering Standards & Citations

• AS 1055:2018: Acoustics — Description and measurement of environmental noise for sensitive receptor buffers.
• NSW EPA Noise Policy for Industry (2017): Industrial noise trigger levels and sleep disturbance criteria (d₀ = 500m).
• ICSM Cadastral Spatial Data Model (CSDM 2020): National and State Cadastral Lot/Plan standardization standard.
• Geoscience Australia ELVIS Elevation Framework: High-resolution DEM slope filtering (ELVIS FSDF ↗).
• Lake Macquarie City Council Economic Development Action Plan: Masterplan clean energy transition strategy (Official PDF ↗).
• Wherobots & Antigravity Engineering Playbook: Enterprise spatial compute, incremental ETL and cost optimization guide (CheatSheets Playbook ↗).

Speed Mechanics

Havasu Spatial Partitioning & Indexing Performance

By leveraging Apache Sedona on Wherobots Cloud with Hilbert-curve spatial partitioning, query scan times across 15.91 million national geometries dropped from 18.4s to 3.2s — an 82% reduction in wall-clock query latency.

1. Hilbert Space-Filling Curve Partitioning

Traditional row-order storage requires full table scans for spatial range queries. Wherobots Havasu re-partitions geometries along a Hilbert space-filling curve so spatially adjacent features land in the same GeoParquet file split. The result:

• Spatial predicate pushdown eliminates irrelevant row groups at the file level — no rows deserialized, no bytes transferred.
• Cadastral join across 15.4M Geoscape parcels: full-table join time 43.1s → 7.8s (Hilbert-indexed vs. row-order).
• ABS meshblock intersect across 368k polygons: 9.2s → 1.6s.

2. GeoParquet Columnar Pushdown

All intermediate and output layers are materialized as GeoParquet with embedded bounding-box statistics per row group. Apache Sedona's ST_Intersects predicate reads only the row groups whose bounding boxes overlap the query polygon — a spatial equivalent of column pruning:

3. Iceberg Snapshot Isolation & Time-Travel

Havasu tables are backed by Apache Iceberg ACID snapshots. Each quarterly cadastral update triggers only an incremental ST_Changes scan rather than a full continental re-ingest:

• Snapshot-level cache validation: The builder fingerprints each Iceberg snapshot ID. If unchanged, the cached GeoParquet layer is reused — zero re-computation.
• Time-travel queries: Engineers can run FOR SYSTEM_VERSION AS OF '2026-08-01' to replay any historical state of the national cadastre.
• Delta partition processing: Only parcels whose geometry changed between snapshot versions are re-processed through the buffer and overlay pipeline.

4. Distributed Executor Sizing

Production headless batch runs use the Wherobots Sedona medium runtime (8 vCPU, 32 GB RAM), which processes the full 15.91M geometry national pipeline in under 4 minutes end-to-end. Right-sizing to the workload — rather than running an over-provisioned cluster idle — is the primary reason the total project compute cost was ~$36 AUD across ~35 batch runs.

What-If Sandbox Mechanics

Real-Time Browser Simulation — Zero Cloud Compute Cost

The What-If Sandbox recalibrates composite MCDA weights and candidate rankings instantly in the browser without any server round-trips, API calls, or cloud compute spend. All recalculation logic runs in-page JavaScript against the precomputed distance and geometry attributes already embedded in this report.

1. Decoupled Architecture: Geometry vs. Scoring

The pipeline is split into two tiers that never need to re-run together:

Changing a weight, threshold, or persona never re-triggers the heavy spatial join pipeline — only the downstream mathematical scoring matrix recalculates, in microseconds, locally.

2. Scoring Formula (Live in Browser)

The composite suitability score uses a four-factor weighted sum:

Each sub-score is a piecewise linear or sigmoidal function of a precomputed distance attribute:

• S_power: 1.0 at 100–500m from substation; linear decay to 0 beyond 5km.
• S_sensitive: Hard exclusion <300m; sigmoidal sigmoid from 300–1500m; plateau 1.0 at 1.5–5km; decay beyond.
• S_water: 1.0 ≤1km from WWTW; linear decay to 0 beyond 10km.
• S_size: 1.0 ≥15 ha; linear ramp from 0.1 at 3 ha; 0.1 below 3 ha.

The weights w_power, w_sensitive, w_water, w_size (summing to 1.0) are controlled by the four sliders in the panel above — or loaded from a stakeholder persona preset.

3. Persona Engine

Selecting a persona from the Strategic Personas tab loads a predefined weight configuration and re-runs the scoring instantly:

Persona weights live in runner/attachments/persona_configs.json — editable without touching any Python or HTML.

4. Slope Exclusion & TSF Toggle

• Slope > 5%: Any site with a DEM slope grade exceeding 5% is hard-excluded (score = 0) regardless of other weights. The ELVIS DEM slope was pre-validated at 25m resolution.
• TSF Toggle: The "Exclude Tailings Storage Facilities" toggle removes sites on or adjacent to TSF footprints from the ranked list. TSF proximity is a binary flag derived from the NSW SEED and EPA spatial layers at build time.

5. Future: DuckDB-WASM Integration

The next evolution is to expose the full GeoParquet candidate layer to a client-side DuckDB-WASM instance embedded in this page. Users will be able to run arbitrary spatial SQL directly in the browser — filtering, grouping, and ranking candidates across all 15.91M national geometries with HTTP range-request byte fetching from GCS — at $0.00 cloud compute cost.

Recent Changes

All active files in docs/ (excluding docs/archive/) have been systematically reviewed one-by-one for currency, link integrity, and technical accuracy. A new author biography document has also been created.

1. Summary of Actions by File

2. Verification Results

• JSON Validation: Verified both .json files via Python json.load() with zero syntax errors.
• Link Integrity: Confirmed all relative and markdown links across docs/ point to active files without broken references.
• Archive Isolation: docs/archive/ was untouched throughout the entire process.
• Compute / Instance Status: All processes and tasks are terminated; zero background compute or active Spark/Sedona sessions are running.

Next Steps: GeoLibre Open-Source AI Spatial Platform Integration

Executive Summary

The primary next step for the National Data Center Siting Suitability Model (hunter_spatial_crafter) is connecting our spatial outputs to an open-source opengeos/GeoLibre deployment hosted serverless on Google Cloud Platform (GCP) to enable free public conversational spatial analytics ("Ask AI") using DuckDB-WASM and Gemini.

1. GeoLibre AI Spatial Platform Architecture on GCP

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                GOOGLE CLOUD PLATFORM (GCP)                              │
│   ┌────────────────────────────────┐                 ┌──────────────────────────────┐   │
│   │ Google Cloud Storage (GCS)     │                 │ GCP Cloud Run (Serverless)   │   │
│   │ • GeoParquet Suitability Data  │◄────────────────┤ • FastAPI Spatial AI Proxy   │   │
│   │ • PMTiles Vector Layers        │  (Direct HTTP)  │ • Scale-to-Zero Container    │   │
│   │ • Static GeoLibre App UI       │                 │ • Gemini & OpenRouter Client │   │
│   └───────────────▲────────────────┘                 └──────────────▲───────────────┘   │
└───────────────────┼─────────────────────────────────────────────────┼───────────────────┘
                    │ static assets & byte-range queries              │ Prompts & AI SQL
                    ▼                                                 ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                END-USER WEB BROWSER                                     │
│   ┌─────────────────────────────────────────────────────────────────────────────────┐   │
│   │ GeoLibre Web Application (Free Web Platform)                                    │   │
│   │  ┌─────────────────────────────┐           ┌─────────────────────────────────┐  │   │
│   │  │ Client-Side DuckDB-WASM     │           │ AI Spatial Chat Drawer          │  │   │
│   │  │ (Zero-Cost In-Browser SQL)  │           │ (Free Tier + BYOK OpenRouter)   │  │   │
│   │  └─────────────────────────────┘           └─────────────────────────────────┘  │   │
│   └─────────────────────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────────────────┘

2. Shared Cloud Data Storage Architecture

Rather than maintaining separate file servers or copying data onto local web server disk storage, GeoLibre and Wherobots share the exact same cloud-native dataset repository:

1. Zero-Duplication Central Data Layer: Wherobots Cloud (Apache Sedona Spark) outputs suitability modeling layers directly into a central GCS / S3 bucket as GeoParquet and PMTiles. GeoLibre queries these exact same files without any dataset conversion or server duplication.
2. HTTP Range-Request Querying: GeoLibre's in-browser DuckDB-WASM engine fetches only the required byte ranges and spatial row groups via HTTP range requests (read_parquet('https://storage.googleapis.com/.../datacenter_candidates.parquet')). Eliminates downloading large files to the client or maintaining expensive local NVMe storage.
3. Cloud-Native Storage Tradeoff:

   Architecture Model	Data Sync / Duplication	Storage & Server Cost	Scalability
   Shared Cloud Storage (GCS)	Zero Duplication (Unified)	Near-Zero (~$0.02/GB/mo)	Infinite Public Scale
   Local Web Server Storage	High Duplication Needed	Expensive Server Disks	Constrained by VM I/O

3. Free Public Conversational AI Workflow ("Ask AI")

1. User Ask: The user types a natural language query: "Show me all candidate sites in VIC larger than 10 hectares that are within 2km of high-voltage transmission lines and at least 1km away from any school or hospital."
2. AI Translation: GeoLibre sends the user prompt + active dataset schemas to the FastAPI gateway on GCP Cloud Run. Cloud Run queries Gemini LLM to generate DuckDB Spatial SQL.
3. In-Browser Compute: The SQL query is executed directly inside the user's browser using DuckDB-WASM, fetching only required byte ranges from GCS.
4. Instant Visualisation: GeoLibre renders matching candidates on an interactive Mapbox/Leaflet map with dynamic stats and spatial boundaries.

4. Dual-Model LLM Access: Free Tier & OpenRouter BYOK

Inspired by GetBack2Basics/LivePersonaCrafter, GeoLibre provides a flexible dual-tier model:

• Free Default Tier: Powered by Google Cloud Gemini API (hosted on Cloud Run), allowing the public to ask natural language questions for free without creating accounts.
• OpenRouter BYOK Tier: Users can enter their own OpenRouter API key directly in the GeoLibre settings drawer to unlock premium models (e.g., Claude 3.5 Sonnet, GPT-4o, DeepSeek-R1, Llama 3) for specialized spatial reasoning.

5. Planned Conversational AI Benchmark Queries

• "Show me all candidate sites in NSW within 2km of 330kV transmission lines that are at least 1km away from schools and child care centers."
• "Which sites in Latrobe Valley or Gladstone score highest for recycled water availability without impacting residential meshblocks?"
• "Find data center parcels co-located within Renewable Energy Zones that have over 15 hectares of developable land."
• "Compare the developable pad area of the Macquarie Coal Complex against the proponent masterplan after deducting 30m riparian buffers and slope constraints."