How AlignCast Generates and Evaluates Forecasting Examples
AlignCast measures whether models can output calibrated probabilities that code will pass tests, without executing code at inference time. This page documents data generation, oracle labeling, split policy, and metrics.
This page documents SpecCast v1.2, in which the forecaster sees the tests. That version is saturated: a zero-shot model answers every test item correctly. The current release, SpecCast v1.4, hides the tests and scores each forecast against the program's true pass probability; its method is documented on the dataset page.
1. Generation
Synthetic Example Construction
Each example includes a specification, an implementation, and deterministic tests. AlignCast v1.2 spans 224 explicit program families over 22 applicable (template, fault family) pairs.
Design Constraints
Python-only, deterministic tests, bounded runtime, no network, and no weaponizable payloads.
Diversity Controls
Seven structural implementation strategies per template/fault pairing, resampled deterministic tests, and near-miss failures. Fault-hint comments are stripped from all evaluation prompts.
We publish high-level fault categories and version IDs alongside the full generator source.
2. Oracle
Ground Truth Labeling
Ground truth is produced by sandboxed test execution. The forecasting model never sees runtime outcomes during inference.
- Per-example timeout and deterministic execution context.
- Recorded outputs: pass/fail label, failing-test count, runtime.
- Network is disabled during oracle execution.
3. Splits
IID and OOD Evaluation Policy
Row-random splits are invalid because multiple rows can resample tests around the same implementation. AlignCast v1.2 groups every partition by program_id.
- IID: program-disjoint train, validation, and test artifacts. The canonical 10k release contains 8,050 / 975 / 975 rows across 180 / 22 / 22 program families.
- v0 limitation: the template-to-fault mapping was 1:1, so holding out a template was equivalent to holding out its fault family — one OOD axis, not two.
- v1.2: a sparse matrix of 22 applicable pairs across 10 templates and 9 stored fault-family labels supports separable template-axis and fault-axis OOD evaluation. The axes are structured, not statistically independent.
- OOD closure: if a held-out row selects a program family, every row sharing that
program_idjoins the test partition. The release includes 10 template-held-out and 9 fault-family-held-out manifests. Fault-family results report the true held-out seed rows and closure-only rows separately, alongside the aggregate.
Only spec, code, and tests enter model context. Evaluation labels are stored separately; public blind Hugging Face splits retain a schema-compatible label column containing only null values. Template, fault, generator, and oracle metadata remain in an analysis sidecar. Split validation independently fingerprints code rather than trusting declared family IDs.
The v1.2 matrix below shows which (template, fault family) pairs are implemented. A cell is filled only where the fault type is semantically natural for that problem — forced pairings are excluded to preserve example quality.
| Structural | Semantic | Boundary | Type / Contract | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| off_by_one | predicate_inversion | wrong_selection | missing_dedup | comparison_weakening | boundary_swap | case_handling | missing_tie_check | silent_error_masking | n | |
| sum_even | 2 | |||||||||
| clamp | 2 | |||||||||
| rotate_left | 2 | |||||||||
| median3 | 2 | |||||||||
| count_vowels | 2 | |||||||||
| unique_sorted | 2 | |||||||||
| find_first_gt | 3 | |||||||||
| majority_vote | 3 | |||||||||
| parse_int_list | 2 | |||||||||
| second_largest | 2 | |||||||||
| ↑ templates | 4 | 3 | 2 | 2 | 3 | 2 | 2 | 2 | 2 | |
4. Metrics
Calibration and Classification Views
Primary metrics are proper scoring-rule calibration metrics, with confusion-matrix diagnostics as a secondary view.
Primary
NLL, Brier, and ECE (lower is better).
Secondary
Thresholded confusion metrics at p_pass >= 0.5: accuracy, precision, recall, F1, TP/FP/TN/FN.
5. Reproducibility
Reference Commands
All published runs should be reproducible from versioned artifacts and deterministic seeds.
6. Safety
Disclosure and Risk Policy
- No real exploit payloads or deployment-ready attack code in benchmark content.
- Public release prioritizes methodological clarity over exploit detail.
- Sensitive deceptive-pattern specifics can be released in stages.