Benchmark Pack Quality

Overview

The spatialpack benchmark run command lets you define quality gates for your spatial packs and verify them after each build. Use it as a post-build CI step to catch regressions (oversized tiles, duplicate IDs, network fragmentation) before they reach production.

When to Use Benchmarks

• Post-build quality gate — run after spatialpack pack build to verify the output meets your standards
• CI regression testing — add to your pipeline to block deploys when quality degrades
• Threshold calibration — run once on a known-good pack to observe actual metric values, then set gates with headroom

Creating Your First .benchmark.yaml

Create a benchmark spec alongside your pipeline YAML:

benchmark_id: wa-roads-z10
description: "Quality gates for WA Road Network at zoom 10"

pmtiles_path: "packs/wa-site-suitability-v2/layers/roads.pmtiles"
pipeline_spec: "pipelines/wa-site-suitability-v2.yaml"  # required for --build

aoi:
  perth_metro: [115.65, -32.15, 116.15, -31.65]

zoom: 10

layers:
  - roads

metrics:
  tile_bytes: true
  feature_count: true
  duplicate_ids: true

gates:
  - name: max_tile_size
    metric: max_compressed_tile_bytes
    operator: "<="
    threshold: 786432    # 768 KB
    severity: critical

See the benchmark CLI reference for the full spec format and available metrics.

Calibrating Thresholds

Don’t guess thresholds. Instead:

1. Run the benchmark without gates (or with very permissive gates) on a known-good pack
2. Check benchmarks/results/<benchmark_id>/<timestamp>/result.json for actual metric values
3. Set your gate thresholds at actual value + ~20% headroom
4. Example: if max_compressed_tile_bytes is 512KB, set the gate at threshold: 614400 (512 × 1.2)

This prevents gates that are too tight (false positives from minor variation) while still catching real regressions.

Running in CI

Use the exit code to gate your deploy:

# Run benchmark -- exits 1 if any critical gate fails
spatialpack benchmark run pipelines/wa-road-z10.benchmark.yaml

# Check exit code
if [ $? -ne 0 ]; then
  echo "Pack quality gates failed -- deploy blocked"
  exit 1
fi

Exit codes:

• 0 — all gates passed
• 1 — one or more critical gates failed
• 2 — runtime error (missing file, load failure)

Result artifacts are written to benchmarks/results/<benchmark_id>/<timestamp>/result.json and include all gate outcomes plus actual metric values.

The --build Flag

Run your build pipeline and benchmark in a single step:

spatialpack benchmark run pipelines/wa-road-z10.benchmark.yaml --build

This builds the pipeline (using pipeline_spec from your benchmark YAML), then immediately runs the benchmark against the fresh output. Useful for local development and CI pipelines where you want a single command.

Road Fragmentation Metric

The road_continuity_ratio metric measures network connectivity:

• Value range: 0.0 to 1.0
• Meaning: Largest connected road component divided by total road nodes
• Ideal: Close to 1.0 (well-connected network)
• Red flags: Values below 0.5 suggest significant fragmentation, likely from clipping or topology errors

Tuning endpoint_tolerance_decimals:

metrics:
  road_fragmentation:
    road_layer: roads
    endpoint_tolerance_decimals: 5  # default: 5 decimal places (~1m tolerance)

Higher values = stricter matching (fewer connections detected). Lower values = more tolerant matching. Start at 5 and tune based on your data’s coordinate precision.

Gate Severity

• critical — pack fails, exit code 1. Use for hard requirements (max tile size, no duplicate IDs).
• warning — advisory only. The benchmark reports the failure but still exits 0. Use for soft targets you’re tracking but not enforcing.