Running the pipeline#

This guide is the runbook: the commands to reproduce everything implemented so far, in the order their inputs require. It complements the pipeline and its cache, which explains the staging and caching design. Here the focus is what to run.

Before the first run#

The pipeline reads the SPARK and SSC releases through the dscat catalogue, so the datasets have to be in place under data/ and the catalogue built once with uv run dscat ingest (see the cohort interface for the releases used). The authors’ feature list and category map have to be present as well. Install the environment with uv sync. The analysis package is a non-editable path dependency, so after editing its source run uv sync --reinstall-package analysis before the change takes effect at the command line.

Every command is uv run analysis <stage>. With no options each stage uses the reference settings: SPARK 2026-03-23, four classes, and 200 restarts.

The stages, in order#

The reference solution comes first, then the stability and replication checks that lean on it.

Build the cohort matrix and the feature typing:
```
uv run analysis cohort
```
fit builds this for you if it is missing, so this step is mainly to inspect the cohort (the proband count, the typing counts, and any typing conflict) before fitting.
Fit the reference mixture model and predict a class label per proband:
```
uv run analysis fit
```
This is the expensive step, around ten minutes for the 200 restarts.
Summarise each class and align the classes to Litman’s named classes:
```
uv run analysis align
```
fit then align are the reference solution. Their result, the per-category profile correlation and the named-class anchors, is the phase-1 reproduction gate (see reproducing the reference classes).
Grid over the number of components and report the information criteria:
```
uv run analysis select
```

Check that the solution survives re-initialisation and resampling:

uv run analysis stability --mode multi-init
uv run analysis stability --mode subsample

Fix the minimum viable stratum size for the later stratified work:
```
uv run analysis nmin
```
Project the model onto the SSC and correlate the category profiles:
```
uv run analysis replicate
```
Characterise the stratification axes and test the binning policies:
```
uv run analysis strata-describe
```

Stages 4 to 7 are the phase-2 stability and replication checks, each reported in its own investigation: selecting the number of classes, stability under refitting, the minimum viable stratum size, and replicating in the SSC. The SSC milestone ages replicate reads are parsed from free text, described in parsing the SSC milestone ages. Stage 8 is the phase-3 feasibility check that fixes the stratification bins, described in choosing the stratification bins.

What depends on what#

fit needs the cohort, and builds it if absent.
align needs a completed fit.
stability and nmin need a completed fit and align (the reference). They exit with guidance if either is missing for the chosen settings.
select and replicate need only the cohort, not the reference, so they can run at any point once the data is in place.
strata-describe needs only the cohort too, and fits no model.

Re-running and the cache#

Each stage writes its outputs and a manifest to artefacts/<stage>/<run-hash>/. The run hash is taken over the stage’s declared inputs (the dataset and version, the feature-list and category-map file digests, the hyperparameters, and the seed), so a later run with the same inputs is a cache hit and returns at once, while a changed input recomputes.

Two consequences are worth keeping in mind:

A change to a stage’s settings, a different release, seed, or restart count, gives a new hash and a fresh run, and leaves the earlier run in place.
A change to the computation that the hash does not capture, for example the SSC milestone-age parsing that replicate applies, does not change the hash. Pass --force to recompute in that case, otherwise the stage returns the cached result from before the change.

A long run that is interrupted does not start over. The multi-seed stages, select, both stability modes, and nmin, checkpoint each completed unit (a seeded iteration, a fit, a replicate) inside the run directory, so re-running the same command resumes from where it stopped and reproduces the same result. --force discards the checkpoint and recomputes from the start.

What is not yet runnable#

The strata, stratify, drift, sensitivity, and report stages are planned for later phases. They appear under “Planned” in uv run analysis --help and exit with a note rather than running. The pipeline implemented so far ends at strata-describe, the phase-3 feasibility check; the stratified fits and their drift metrics come next.