Training Guide

This guide covers the trainer loop and how to combine SCM semantics with coverage-aware optimisation and named epoch-boundary resume profiles.

Trainer Overview

  • zeroproofml.training.trainer.SCMTrainer implements the reference loop with mixed precision, gradient accumulation, and coverage-based early stopping.
  • zeroproofml.training.targets.lift_targets is a deprecated convenience for explicitly converting sentinel-encoded data outside the trainer. The trainer never interprets NaN/Inf in an ordinary two-item (inputs, values) batch: that form is finite-only and rejects non-finite values.
  • For auditable datasets with explicit label semantics, prefer zeroproofml.training.targets.lift_semantic_targets(values, status_labels) with status labels from SemanticTargetLabel: FINITE, BOTTOM, CENSORED_BELOW, CENSORED_ABOVE, EXACT_DOMAIN_INVALID, OBSERVATION_DOMAIN_INVALID, MISSING, or FAULT. Raw string labels are rejected rather than inferred. The helper returns SemanticTargets carrying (Y_n, Y_d), finite/bottom masks, censored orientation labels, and bottom-kind codes that distinguish semantic bottoms from faults.
  • SCMTrainer accepts typed batches as (inputs, values, status_labels) and passes a semantic_targets keyword argument to loss functions that opt into it through an explicit parameter. A generic **kwargs parameter does not count as semantic-target support. Losses should use that object to mask labels such as MISSING, which are neither finite training targets nor bottom targets.
  • Thresholds are perturbed per batch (perturbed_threshold) to reduce train/infer gaps.

TrainingConfig

zeroproofml.training.trainer.TrainingConfig controls the trainer loop:

  • Epochs/updates: max_epochs (>= 1), gradient_accumulation_steps (>= 1, validated at construction). Since v0.6.1 a trailing partial group at epoch end is flushed automatically and scaled by its actual group size, not the configured accumulation count.
  • AMP: mixed_precision (alias: use_amp) and amp_dtype. v0.6.1 uses the unified torch.amp API when available and falls back to the legacy torch.cuda.amp / torch.cpu.amp paths needed by the advertised torch>=1.12 support floor.
  • Thresholds: tau_train_min, tau_train_max, and strict tau_infer. All three go through the shared v0.6.1 validators — tau_infer must be finite and > 0; tau_train_* must be finite, positive, and >= tau_infer; tau_train_min <= tau_train_max is enforced too. Any violation raises ValueError at TrainingConfig(...) construction.
  • Coverage early-stop: coverage_threshold is a finite fraction in [0, 1]; coverage_patience is a positive integer. Early stopping reads the validation aggregate (val_history[-1]["val_coverage"]), not the last training-batch metric, and stops after exactly that many consecutive below-threshold validation epochs.
  • Scheduler cadence: scheduler_cadence ∈ {"per_batch", "per_optimizer_update", "per_epoch", "per_validation_metric"} (defaults to "per_optimizer_update" to preserve pre-v0.6.1 behavior).
  • Logging: log_hook(metrics) (see 15_debug_logging.md)
  • Validation: val_loader runs once per epoch; aggregated metrics are stored in trainer.val_history and emitted to log_hook with val_-prefixed keys. Aggregation is sample-count-weighted across batches so unequal validation batch sizes produce the same aggregate as a flat sample-level pass; per-key custom reducers (min / max / ratio / confusion) can be registered on trainer.metric_reducers.
  • Gradient policy override: gradient_policy applies a global GradientPolicy override during training steps (see 03_gradient_policies.md).
  • Projective representation: projective_coordinate_axis="model" reads a model's declared tuple axis (the shared-denominator rational heads declare -1). Set an integer explicitly for another shared-denominator tuple, or None for elementwise scalar pairs. The resolved value is load-bearing checkpoint configuration.
  • Complex tensors: projective normalization, training, and typed strict decoding are real-valued. Complex coordinates raise TypeError; use the separately named phase API for complex unit phase.
  • Bottom capability check: if typed targets contain SemanticTargetLabel.BOTTOM and a projective head reports bottom_capability(tau_infer) == "unreachable_by_construction", the trainer raises before optimizing. Use allow_bottom_unreachable=True only when those labels are intentional noise or outside the current task. Note: tau_infer = 0 no longer counts as unreachable-by-construction; that fail-open path is refused up-front by the strict-threshold validator.
  • Loss curricula (optional): loss_curriculum can produce per-epoch loss_weights that are passed into loss functions that accept loss_weights (and epoch / global_step).
  • Optimizer and checkpoint contract: optimizer parameter-group learning rates are authoritative; TrainingConfig.learning_rate remains compatibility metadata. save_checkpoint(path) writes schema v4 with model, optimizer, scaler, scheduler, progress, early-stop state, semantic identity, named resume profile, required RNG/sampler state, actual parameter-group rates, and separate hashed resume/operational configuration identities. Drift in a load-bearing field fails; max_epochs may only be extended and produces a structured drift record. Older files load as legacy-unqualified.
  • Resume profile: select resume_profile=ResumeProfile.CPU_BASIC_V1, CPU_SHUFFLED_SINGLEWORKER_V1, CUDA_SINGLE_DEVICE_V1, or UNQUALIFIED. Strong profiles validate device, worker, distributed, sampler/generator, and deterministic-algorithm preconditions. The shuffled CPU profile supports RandomSampler or a custom sampler with state_dict() / load_state_dict(mapping) plus an explicit torch.Generator.
  • Checkpoint trust: load_model_state(path) and load_checkpoint(path) are fail-closed untrusted loaders. They require PyTorch 2.10.0 or newer and weights_only=True; older releases raise UnsafeCheckpointLoadUnavailable before deserialization. An optional expected_sha256 binds the exact artifact. load_trusted_checkpoint(path) is the warned pickle-capable exception and must never receive an untrusted artifact. See the checkpoint/resume contract.

Typical Flow

  1. Prepare data with SemanticTargetLabel status labels whenever any target is non-finite or non-ordinary; the trainer lifts (inputs, values, status_labels) batches to (Y_n, Y_d). Use a two-item batch only when every value is an ordinary finite target.
  2. Select gradient policy (usually CLAMP for SCM-only graphs or PROJECT for projective heads).
  3. Assemble losses: implicit + margin + sign consistency + rejection (via SCMTrainingLoss).
  4. Train loop:
  5. forward pass (SCM or projective mode),
  6. compute losses and coverage,
  7. backprop using the active gradient policy,
  8. update optimiser (supports AMP through torch.amp when available, with legacy AMP fallbacks for the torch>=1.12 compatibility floor).
  9. Monitor coverage; early stop when coverage stays below coverage_threshold for coverage_patience epochs.

Tips

  • Trainer coverage normalizes (P, Q) with the canonical gauge; do not derive coverage from raw abs(Q) or treat a non-finite payload as algebraic bottom.
  • Pass coordinate_axis=-1 to soft_coverage_loss for a vector numerator with one shared denominator. Elementwise rational pairs keep the default None.
  • SCMTrainingLoss derives its margin from the canonical (P,Q) score. The direct Q-only margin and separation helpers are retained for v0.6 compatibility; pass numerator=P (and the tuple axis when applicable) in maintained 0.7 paths. Adaptive-sampler inputs are canonical denominator scores in [0,1], not raw network denominators.
  • Keep τ_train_min and τ_train_max close unless you specifically need stronger perturbations.
  • Log last_thresholds from the trainer to understand how often the model sees near-singular regimes.