Test strategy and evidence tiers¶

Tests are classified by purpose even when they share the top-level tests/ tree. Markers express the execution tier; a new test should use the most specific applicable marker and location.

Class	Marker/location	Purpose
Unit	subsystem directories under tests/	Local behavior and invariants
Integration	integration	Multi-component/backend workflows
Regression	regression	A previously observed, reproducible defect
Property	property	Generated or metamorphic invariants
Compatibility	compatibility	Supported dependencies, platforms, and file formats
Golden	golden	Small, reviewed deterministic behavior snapshots
Verification bridge	verification	Python/schema/checker/formal interfaces
Performance	benchmark, perf/	Timing/memory history, not normal unit tests
Scientific experiment	experiments/	Research hypothesis evaluation, not a unit test

Every reproducible bug fix adds a regression test. Randomized tests use explicit seeds or deterministic generators; floating-point assertions state a mathematically justified tolerance and, if needed, a platform scope. Golden baseline updates must identify the generating command, declared environment, and the reason the changed result is scientifically correct.

scripts/ci/test_profiles.py is the executable source of truth for every release profile. make test-fast resolves mr-fast; make test and make test-coverage resolve tag-core; make test-min-torch resolves the explicit lower-bound minimum-torch compatibility profile. Those base profiles exclude dependency-backed success paths. make test-onnx, make test-jax, and make test-viz install and execute their named optional profiles; make test-paper-contract verifies the frozen 0.4.3 replay boundary, and make test-scientific-contract verifies the bounded current benchmark contract. Every profile must collect at least one node and its collection manifest contains a sorted, hashed node list. Profile manifests also record the path-universe count and marker deselection count, so a release baseline distinguishes a deliberately filtered profile from silently lost collection nodes without parsing pytest's human summary.

make test-all runs the full base/optional/package profile set in its declared environment. The separately provisioned lower-bound minimum-torch profile is run with make test-min-torch after installing its declared Torch version. For a frozen candidate that already has reviewed baselines, run make test-coverage VERIFY_RELEASE_RATCHETS=1 QUALIFICATION_COMMIT=<C>; this same canonical command produces the evidence and then blocks on exact test and coverage ratchets. Do not enable that switch against a moving development tree or use it to generate a baseline. make test-release-blockers is a supplemental fail-closed sweep of every contract found red by the v0.6.1 audit; it emits a JUnit report and exact node manifest but does not replace the complete profile matrix. Distribution smoke tests run through make build or make check-dist. Long experiments and performance benchmarks use their own commands/manifests and must not silently become part of an ordinary unit-test selector. The optional scripts/local_ci.sh wrapper delegates only to these Make targets and preserves their exit statuses while retaining local logs.