Checkpoint security and resume contract¶
Checkpoint deserialization and deterministic resume are separate capabilities. Loading model tensors safely does not by itself establish that a training trajectory can be reproduced, and a byte digest does not make an untrusted pickle producer safe.
Deserialization capability matrix¶
| API | Torch requirement | Restored state | Trust requirement |
|---|---|---|---|
load_checkpoint_mapping(...) |
PyTorch 2.10.0+ with weights_only |
validated bounded mapping | untrusted artifact allowed |
SCMTrainer.load_model_state(...) |
PyTorch 2.10.0+ with weights_only |
model only | untrusted artifact allowed |
SCMTrainer.load_checkpoint(...) |
PyTorch 2.10.0+ with weights_only |
schema-qualified resume state | untrusted artifact allowed |
SCMTrainer.load_trusted_checkpoint(...) |
general advertised Torch floor | full/legacy resume state | producer and custody must be trusted; warns |
load_checkpoint_mapping(..., trusted=True) |
general advertised Torch floor | mapping, possibly through pickle | producer and custody must be trusted |
The general arithmetic and trainer compatibility floor remains
torch>=1.12. ZeroProofML refuses every untrusted checkpoint API below
PyTorch 2.10.0, even if that release exposes a weights_only keyword. Releases
through 2.9.1 are affected by the upstream weights-only memory-corruption
advisory. An unavailable, old, or unparseable capability raises
UnsafeCheckpointLoadUnavailable before torch.load is called; there is no
unsafe fallback.
| Capability | Declared minimum |
|---|---|
| Core SCM | no third-party runtime dependency |
| Torch arithmetic/training | PyTorch 1.12 |
| ONNX deployment extra | PyTorch 2.7.1 plus the pinned ONNX dependencies |
| Untrusted checkpoint loading | PyTorch 2.10.0 |
Installing the deployment extra on PyTorch 2.7.1 through 2.9.1 does not enable untrusted checkpoint loading; the higher checkpoint floor still applies.
All public checkpoint entry points enforce a file-size bound, optional expected
SHA-256, bounded field/key/container depth and count, bounded tensor storage,
and supported value types. The loader copies the bounded input into a private
snapshot while hashing and gives that same snapshot to torch.load, so path
replacement cannot substitute different bytes after digest verification.
Schema-v4 resume validates its complete field set, digests, component
contracts, model keys/shapes/dtypes, optimizer structure, profile, and resume
state before mutating the model, optimizer, scheduler, scaler, counters, or
RNG. Component application is transactional and restores package-owned state
if a component-specific loader still raises. expected_sha256 should come
from an authenticated external manifest; a digest stored beside an
attacker-controlled file does not authenticate it.
Format policy¶
Checkpoint schema v4 is the qualified package resume format. It stores model, optimizer, scheduler, scaler, progress, early-stop state, semantic identity, the named resume profile, captured RNG/sampler state, component contracts, and separate resume-load-bearing and operational configuration identities. Unknown or missing v4 fields fail closed.
Schema v0-v3 files use an explicit compatibility reader. They restore only the
state representable in their schema and are labelled legacy-unqualified;
they never inherit a deterministic-resume claim. Use the trusted loader if a
legacy file contains objects not admitted by safe weights-only loading.
Model-only interchange may use a bounded mapping containing model and an
optional schema marker. For independent cross-tool interchange or stronger
resource isolation, use a non-pickle tensor format outside the package
checkpoint contract and adapt its validated mapping to load_state_dict.
Named resume profiles¶
Resume is supported only at completed epoch boundaries.
| Profile | Preconditions | State restored | Claim |
|---|---|---|---|
cpu-basic-v1 |
CPU, num_workers=0, SequentialSampler, no external RNG |
Torch CPU RNG and all v4 component/progress state | same trajectory under identical load-bearing contract |
cpu-shuffled-singleworker-v1 |
CPU, num_workers=0, explicit torch.Generator, RandomSampler or custom stateful-sampler protocol |
CPU RNG, loader generator, sampler reconstruction/state, all v4 state | same shuffled epoch-boundary trajectory |
cuda-single-device-v1 |
first CUDA device, deterministic algorithms enabled, num_workers=0, no distributed training, built-in sequential/random sampler |
CPU and every visible CUDA RNG, optional loader generator/sampler state, all v4 state | same trajectory only on the qualified CUDA stack |
unqualified |
none beyond ordinary trainer validity | available state only | no deterministic-resume claim |
The custom shuffled-sampler protocol is deliberately small: expose callable
state_dict() and load_state_dict(mapping), expose or share the explicit
loader torch.Generator, and make the state sufficient to reconstruct the next
epoch's order. Unsupported custom, multi-worker, distributed, mid-epoch, or
data-stream environments fail before a strong profile is used.
Named profiles require the model, loss, dataset, transforms, callbacks, and
sampler not to consume Python random, NumPy RNG, hidden generators, wall
clock, network, or other external mutable state. Those sources are not captured
by these profiles. If they are required, select unqualified until a new
versioned profile records and restores them.
Configuration identity and drift¶
Resume-load-bearing identity includes semantic identity, model state schema, optimizer type/defaults/groups and initial rates, scheduler type/static parameters, loss callable identity, batch/order/sampler contract, generator initial seed, gradient accumulation, precision, thresholds, gradient policy, device/profile, loss weights/curriculum, scheduler cadence, and projective axis. Drift in any such field rejects resume.
max_epochs is operational. It may be extended but not reduced; every accepted
extension produces a structured last_resume_drift record. No other drift is
currently permitted.
Qualification boundary¶
Local tests cover CPU basic and shuffled interruption/resume, custom sampler state, schema tampering, malicious pickle refusal, and configuration drift. CUDA equivalence, the complete Python/OS/dependency matrix, and immutable candidate-artifact qualification remain release gates. Passing local tests is not a deployment or universal reproducibility claim.
See SECURITY.md and TRUST_BOUNDARIES.md for the broader threat model.