Public Methodology

How AlignCast Generates and Evaluates Forecasting Examples

AlignCast measures whether models can output calibrated probabilities that code will pass tests, without executing code at inference time. This page documents data generation, oracle labeling, split policy, and metrics.

This page documents SpecCast v1.2, in which the forecaster sees the tests. That version is saturated: a zero-shot model answers every test item correctly. The current release, SpecCast v1.4, hides the tests and scores each forecast against the program's true pass probability; its method is documented on the dataset page.

Taxonomy reference: fault_taxonomy_v1.md


1. Generation

Synthetic Example Construction

Each example includes a specification, an implementation, and deterministic tests. AlignCast v1.2 spans 224 explicit program families over 22 applicable (template, fault family) pairs.

Design Constraints

Python-only, deterministic tests, bounded runtime, no network, and no weaponizable payloads.

Diversity Controls

Seven structural implementation strategies per template/fault pairing, resampled deterministic tests, and near-miss failures. Fault-hint comments are stripped from all evaluation prompts.

We publish high-level fault categories and version IDs alongside the full generator source.


2. Oracle

Ground Truth Labeling

Ground truth is produced by sandboxed test execution. The forecasting model never sees runtime outcomes during inference.


3. Splits

IID and OOD Evaluation Policy

Row-random splits are invalid because multiple rows can resample tests around the same implementation. AlignCast v1.2 groups every partition by program_id.

Only spec, code, and tests enter model context. Evaluation labels are stored separately; public blind Hugging Face splits retain a schema-compatible label column containing only null values. Template, fault, generator, and oracle metadata remain in an analysis sidecar. Split validation independently fingerprints code rather than trusting declared family IDs.

The v1.2 matrix below shows which (template, fault family) pairs are implemented. A cell is filled only where the fault type is semantically natural for that problem — forced pairings are excluded to preserve example quality.

Structural Semantic Boundary Type / Contract
off_by_one predicate_inversion wrong_selection missing_dedup comparison_weakening boundary_swap case_handling missing_tie_check silent_error_masking n
sum_even 2
clamp 2
rotate_left 2
median3 2
count_vowels 2
unique_sorted 2
find_first_gt 3
majority_vote 3
parse_int_list2
second_largest2
↑ templates 432232222
applicable pair (22 total)
not coded
Structural
Semantic
Boundary
Type / Contract

Full design rationale: docs/fault_matrix_v1.html


4. Metrics

Calibration and Classification Views

Primary metrics are proper scoring-rule calibration metrics, with confusion-matrix diagnostics as a secondary view.

Primary

NLL, Brier, and ECE (lower is better).

Secondary

Thresholded confusion metrics at p_pass >= 0.5: accuracy, precision, recall, F1, TP/FP/TN/FN.


5. Reproducibility

Reference Commands

All published runs should be reproducible from versioned artifacts and deterministic seeds.

python scripts/generate_benchmark.py --n 10000 --seed 42 --out bench/bench_v1_2_10000_seed42.jsonl python scripts/create_splits.py --input bench/bench_v1_2_10000_seed42.jsonl --out-dir bench/splits_v1_2_seed42 python scripts/create_ood_manifests.py --input bench/bench_v1_2_10000_seed42.jsonl --out-dir bench/ood_v1_2_seed42 python scripts/check_leakage.py --split-dir bench/splits_v1_2_seed42 python scripts/check_leakage.py --ood-dir bench/ood_v1_2_seed42

Recorded metadata includes generator_version, oracle_version, sandbox_version, split seeds, and artifact hashes.


6. Safety

Disclosure and Risk Policy