Classification & non-i.i.d. data

V0.1.6: © Aleksei Tiulpin, PhD, 2025

This notebook shows an end-to-end example on how one can compare two models on a test set where data has block-diagonal covariance (i.e. non-i.i.d. samples measured e.g. from the same patient)

Imports

import stambo
import numpy as np

import matplotlib.pyplot as plt

from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedGroupKFold

SEED = 2025

stambo.__version__

'0.1.6'

Generating synthetic data with block-diagonal covaraince

We use stambo’s synthetic dataset generation functionality to generate the dataset in question.

np.random.seed(SEED)
n_subjects = 100
n_data = 300

data, y, subject_ids = stambo.synthetic.generate_non_iid_measurements(
    n_data=n_data,
    n_subjects=n_subjects,
    rho=0.8,             # Subject correlation between classes
    subj_sigma=2.0,       # Strong subject-level effect relative to noise -> more within-subject correlation
    noise_sigma=0.5,
    gamma=0.8,
    mu_cls_1=[2, 2],
    mu_cls_2=[2.1, 2.4],
    overlap=0.3,          # More subjects appear in both classes -> more repeated/correlated measurements
    feat_corr=0.5,       # <--- High correlation between Dim 1 and Dim 2
    seed=2025
)

plt.figure(figsize=(4, 4))
plt.plot(data[y == 0, 0], data[y == 0, 1], "bo")
plt.plot(data[y == 1, 0], data[y == 1, 1], "ro")
plt.show()

gss = StratifiedGroupKFold(n_splits=3, shuffle=True, random_state=SEED)
train_idx, test_idx = next(gss.split(data, y, groups=subject_ids))

# Split the data into training and testing sets
X_train, X_test = data[train_idx], data[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
groups_train, groups_test = subject_ids[train_idx], subject_ids[test_idx]

# Initialize classifiers
knn = KNeighborsClassifier(n_neighbors=5)
logreg = LogisticRegression()

# Train classifiers
knn.fit(X_train, y_train)
logreg.fit(X_train, y_train)

# Make predictions
knn_predictions = knn.predict_proba(X_test)[:, 1]
logreg_predictions = logreg.predict_proba(X_test)[:, 1]

What we should expect is that on a small test set with correlated, repeated measurements from the same subjects, a naive bootstrap (that treats every row as an independent measurement) can make us overconfident: it can report a statistically significant difference between two models even when the evidence, once we account for the subject-level structure, is not actually strong enough to support that conclusion. Let’s look at the naive bootstrap first:

testing_result = stambo.compare_models(y_test, logreg_predictions, knn_predictions, ("ROCAUC", "AP"),seed=SEED, n_bootstrap=1000)
print(stambo.to_latex(testing_result, m1_name="logreg", m2_name="kNN"))

% \usepackage{booktabs} <-- do not forget to have this imported.
\begin{tabular}{lll} \\
\toprule
\textbf{Model} & \textbf{ROCAUC} & \textbf{AP} \\
\midrule
logreg & $0.31$ [$0.21$-$0.42$] & $0.40$ [$0.31$-$0.53$] \\
kNN & $0.47$ [$0.36$-$0.58$] & $0.51$ [$0.40$-$0.64$] \\
\midrule
Effect size & $0.16$ [$0.03$-$0.29]$ & $0.12$ [$0.02$-$0.20]$ \\
\midrule
$p$-value & $0.01$ & $0.02$ \\
\bottomrule
\end{tabular}

We observe a statistically significant result here (both ROC-AUC and AP have a two-tailed p-value below 0.05). Let us now correct for clustering by passing the subject groups to compare_models:

testing_result = stambo.compare_models(y_test, logreg_predictions, knn_predictions, metrics=("ROCAUC", "AP"), groups=groups_test, seed=SEED, n_bootstrap=1000)
print(stambo.to_latex(testing_result, m1_name="LR", m2_name="kNN"))

% \usepackage{booktabs} <-- do not forget to have this imported.
\begin{tabular}{lll} \\
\toprule
\textbf{Model} & \textbf{ROCAUC} & \textbf{AP} \\
\midrule
LR & $0.31$ [$0.15$-$0.56$] & $0.40$ [$0.20$-$0.74$] \\
kNN & $0.47$ [$0.25$-$0.70$] & $0.51$ [$0.32$-$0.72$] \\
\midrule
Effect size & $0.16$ [$-0.18$-$0.43]$ & $0.12$ [$-0.13$-$0.28]$ \\
\midrule
$p$-value & $0.68$ & $0.65$ \\
\bottomrule
\end{tabular}

Summary: According to the naive bootstrap (which ignores the fact that several test rows come from the same subject), the difference between the two models looks statistically significant for both metrics. Once we take the subject-level structure into account with the clustered (cluster/group) bootstrap, the confidence intervals widen considerably and now include 0, and neither p-value is anywhere near the usual 0.05 threshold. In other words, the naive bootstrap was overconfident purely because it ignored the non-i.i.d. structure of the data – this is exactly the failure mode that stambo’s groups argument is designed to correct.