This chapter specifies an evaluation design for comparing the three families. The requirements are stated so that a comparison conducted under them supports interpretable conclusions.
Prediction target and baselines
Fix one target (Chapter 2), normally change or log-ratio, and evaluate all arms on it. Report against three baselines:
- No change. Predict \(y_{i,j+1,r} = y_{ijr}\).
- Population mean change, conditioned on age, sex, and interval.
- Linear extrapolation of the observed rate, where \(T \geq 3\).
Report \(R^2\) or explained variance relative to the no-change baseline, not relative to the grand mean of the target. The latter credits a model for predicting that volumes are approximately what they were, which is not the quantity of interest. A model that does not outperform baseline 1 has not demonstrated any capacity to predict change.
Data splitting
- Split at the subject level. Scan-level splitting places the same subject in training and test and invalidates the evaluation.
- Stratify by site, or use leave-one-site-out where generalization across acquisition conditions is claimed.
- Use identical folds across arms and across regions. Differences between arms must not be confounded with fold assignment.
- Build longitudinal processing templates without held-out visits (Chapter 4).
Primary evaluation design
Where \(T \geq 3\): train on the interval from visit 1 to visit 2 and predict visit 3. This is the primary result. It tests extrapolation, which is where the families differ by construction, and it is the design that prevents the collapse described in Chapter 3.
Report interpolation performance separately. Conflating the two obscures the main finding, which is typically that the arms rank differently under the two.
Where \(T = 2\) only: see Chapter 9 for the available alternatives. A within-interval cross-validation is not a substitute.
Metrics
Report all of:
- Point accuracy per region: mean absolute error and root mean squared error on the target scale.
- Explained variance relative to the no-change baseline, per region.
- Calibration. Empirical coverage of nominal prediction intervals. Mechanistic and hierarchical Bayesian arms frequently tie or lose on point accuracy while winning on calibration; this is a substantive result and should be reported as such rather than omitted.
- Stratification by signal-to-noise ratio (Chapter 4). Aggregate metrics over all regions are dominated by regions in which no method can succeed.
- A downstream utility measure, where one exists — for example, whether predicted rates classify subsequent clinical status better than current values.
Model ladder
Arms should be nested where possible so that performance differences are attributable to identifiable additions rather than to wholesale differences in implementation.
| 0 |
No change |
— |
| 1 |
Population mean change |
Covariate adjustment |
| 2 |
D1 exponential mixed model |
Mechanistic form, random effects |
| 3 |
+ D4 latent time |
Stage alignment |
| 4 |
+ D3 network coupling |
Structural prior |
| 5 |
ML on change, pooled regions |
Flexible cross-sectional mapping |
| 6 |
ML autoregressive |
Propagation |
| 7 |
H2 learned parameterization |
Personalized mechanism |
| 8 |
H1 learned residual |
Correction term |
| 9 |
H3 universal DE |
Learned vector field |
| 10 |
Ensemble of 4, 6, 7 |
— |
Report every step. Intermediate steps that do not improve performance are informative and should not be omitted from the final table.
Ablations
For each hybrid arm, report the ablation that removes the learned component. If H3 does not outperform its own mechanistic skeleton, the neural term is fitting noise and this should be stated.
For the supervised arm, report performance with and without the same-region baseline feature, to quantify the regression-toward-the-mean contribution (Chapter 6).
Power and the noise floor
Before running the comparison, compute the expected effect size per region from the measurement floor and prior estimates of change. State in advance which regions the study is powered to distinguish methods on. Restricting the primary analysis to those regions, with the remainder reported separately, is preferable to reporting an aggregate diluted by uninformative regions.
Reporting
State explicitly: the objective from Chapter 2 being addressed; the target; the baselines; the splitting scheme; the number of subjects retained at each visit; the measurement floor estimate and its source; the latent-time anchoring choice where applicable; and the shrinkage diagnostic for any subject-level parameter that is interpreted.