Main functionality

We perform a two-tailed bootstrap test, comparing two samples using pre-defined statistics. The library works so that the user provides a function that computes the statistic of interest, and predictions of two models on a test set.

stambo.compare_models(y_test, preds_1, preds_2, metrics, groups=None, alpha=0.05, n_bootstrap=5000, seed=None, silent=False)[source]

Compares predictions from two models \(f_1(x)\) and \(f_2(x)\) that yield prediction vectors \(\hat y_{1}\) and \(\hat y_{2}\) with a two-tailed bootstrap hypothesis test.

I.e., we state the following null and alternative hypotheses:

\[ \begin{align}\begin{aligned}H_0: M(y_{gt}, \hat y_{1}) = M(y_{gt}, \hat y_{2})\\H_1: M(y_{gt}, \hat y_{1}) \neq M(y_{gt}, \hat y_{2}),\end{aligned}\end{align} \]

where \(M\) is a metric, \(y_{gt}\) is the vector of ground truth labels, and \(\hat y_{i}, i=1,2\) are the vectors of predictions for model 1 and 2, respectively. Such kind of testing is performed for every specified metric.

Since the test is two-tailed, the \(p\)-value does not depend on which model is passed as model 1 vs. model 2, or on whether the metric is defined as more-is-better or less-is-better. The sign of the reported effect size (\(M(y_{gt}, \hat y_{2}) - M(y_{gt}, \hat y_{1})\)) tells you which model scored higher on the metric as passed.

While the test does return you the \(p\)-value, one should be careful about its interpretation: the \(p\)-value is the probability of observing the test statistic at least as extreme as the one obtained assuming that \(H_0\) is true (probability of Type II error). With large data, even small effects can be statistically significant, so one should consider the effect size.

We compute a standardized effect size using the estimated bootstrap variance.

Beyond the hypothesis testing, the function also returns confidence intervals per metric, i.e.

\[P\left(M(y_{gt,*}, \hat y_*) \in [L_{CI}(\alpha), H_{CI}(\alpha)]\right) = 1 - \alpha,\]

where \(L\) and \(H\) are the lower and upper bounds of the confidence interval, respectively, and \(\alpha\) is the significance level, and \(*\) indicates that the metric is computed on infinite data.

At this moment, the confidence intervals are computed using the simple percentile method. In the future, we will implement the BCa approach, which is more accurate.

Parameters:

y_test (NDArray[int64] | NDArray[float64]) – Ground truth.
preds_1 (NDArray[int64] | NDArray[float64]) – Prediction from model 1.
preds_2 (NDArray[int64] | NDArray[float64]) – Prediction from model 2.
metrics (Tuple[str | Metric]) – A set of metrics to call. Here, the user either specifies the metrics available from the stambo library (stambo.metrics), or adds an instance of the custom-defined metrics.
groups (NDArray[int64] | None) – Groups indicating the subject for each measurement. Defaults to None.
alpha (float) – A significance level for confidence intervals (from 0 to 1). Defaults to 0.05.
n_bootstrap (int) – The number of bootstrap iterations. Defaults to 5000.
seed (int | None) – Random seed. Defaults to None.
silent (bool) – Whether to execute the function silently, i.e. not showing the progress bar. Defaults to False.

Returns:

A dictionary containing a tuple with the empirical value of the metric, and the two-tailed p-value. The expected format in the output in every dict entry is:

Two-tailed \(p\)-value

Observed difference (effect size)

Effect size CI low

Effect size CI high

\(M(y_{gt}, \hat y_{1})\)

\(M(y_{gt}, \hat y_{1})_{(\alpha / 2)}\)

\(M(y_{gt}, \hat y_{1})_{(1 - \alpha / 2)}\)

\(M(y_{gt}, \hat y_{2})\)

\(M(y_{gt}, \hat y_{2})_{(\alpha / 2)}\)

\(M(y_{gt}, \hat y_{2})_{(1 - \alpha / 2)}\)

Return type:

Dict[str, NDArray[float64]]

stambo.compare_models_pairwise(y_test, preds, metrics, labels=None, groups=None, alpha=0.05, n_bootstrap=5000, correction='holm', seed=None, silent=False)[source]

Compares predictions from \(N \geq 2\) models pairwise, i.e. the many-model generalization of compare_models().

For \(N\) models there are \(N (N - 1) / 2\) pairwise comparisons per metric. This function is a thin wrapper that builds a PredSampleWrapper for each model’s predictions (exactly like compare_models() does for two models) and then runs pairwise_bootstrap_test() across all of them. Because testing many pairs inflates the family-wise Type I error rate, the p-values are Holm-Bonferroni corrected per metric by default (correction="holm"); pass correction=None to disable this.

Parameters:

y_test (NDArray[int64] | NDArray[float64]) – Ground truth.
preds (Tuple[NDArray[int64] | NDArray[float64], ...]) – A tuple of prediction arrays, one per model, all evaluated on the same y_test.
metrics (Tuple[str | Metric]) – A set of metrics to call. Here, the user either specifies the metrics available from the stambo library (stambo.metrics), or adds an instance of the custom-defined metrics.
labels (Tuple[str, ...] | None) – Labels for the models, used to name each comparison as "{label_i} / {label_j}". Defaults to ("0", "1", ..., str(N-1)).
groups (NDArray[int64] | None) – Groups indicating the subject for each measurement. Defaults to None.
alpha (float) – A significance level for confidence intervals (from 0 to 1). Defaults to 0.05.
n_bootstrap (int) – The number of bootstrap iterations. Defaults to 5000.
correction (str | None) – Multiple-comparison correction to apply to the p-values, or None to disable it. Only "holm" is currently supported. Defaults to "holm".
seed (int | None) – Random seed. Defaults to None.
silent (bool) – Whether to execute the function silently, i.e. not showing the progress bar. Defaults to False.

Returns:

a dictionary keyed by metric tag, then by comparison label, holding a dict with keys p_value, p_value_adjusted, diff, ci_es, ci_s1, ci_s2, emp_s1, and emp_s2.

Return type:

Same format as pairwise_bootstrap_test()

stambo.two_sample_test(sample_1, sample_2, statistics, groups=None, alpha=0.05, n_bootstrap=5000, seed=None, non_paired=False, silent=False)[source]

Compares whether the empirical difference of statistics computed on two samples is statistically significant or not.

The hypotheses we test are:

\[\begin{split}H_0: f(x_1) = f(x_2) \\ H_1: f(x_1) \neq f(x_2),\end{split}\]

where \(f\) is a function of interest, and \(x_1\) and \(x_2\) are the samples to be compared. Note that the statistics are computed independently, and should thus be treated independently.

Parameters:

sample_1 (NDArray[int64] | NDArray[float64] | PredSampleWrapper) – Sample 1 to be compared.
sample_2 (NDArray[int64] | NDArray[float64] | PredSampleWrapper) – Sample 2 to be compared.
groups (NDArray[int64] | None) – Groups indicating the subject for each measurement. Defaults to None.
statistics (Mapping[str, Callable]) – Statistics to compare the samples by.
alpha (float) – A significance level for confidence intervals (from 0 to 1).
n_bootstrap (int) – The number of bootstrap iterations. Defaults to 5000.
non_paired (bool) – Whether to use a non-paired design. Defaults to False.
seed (int | None) – Random seed. Defaults to None.
silent (bool) – Whether to execute the function silently, i.e. not showing the progress bar. Defaults to False.

Returns:

A dictionary containing a tuple with the empirical value of the metric, and the p-value. Each entry in the dictionary contains, in order:

Two-tailed \(p\)-value, \(p(\texttt{data} \mid H_0)\)

Observed difference (effect size)

CI low (effect size)

CI high (effect size)

Empirical value (sample 1)

CI low (sample 1)

CI high (sample 1)

Empirical value (sample 2)

CI low (sample 2)

CI high (sample 2)

Return type:

Dict[str, NDArray[float64]]

stambo.pairwise_bootstrap_test(samples, statistics, groups=None, labels=None, bootstrap_results=None, alpha=0.05, n_bootstrap=5000, correction='holm', seed=None, silent=False)[source]

Runs the two-tailed bootstrap test from two_sample_test() on every pair among \(N \geq 2\) mutually paired samples (e.g. \(N\) models evaluated on the same test set).

For \(N\) samples there are \(N (N - 1) / 2\) pairwise comparisons. Testing many pairs inflates the family-wise Type I error rate (the more pairs you test, the more likely some pair looks “significant” purely by chance), so by default (correction="holm") a Holm-Bonferroni correction (see holm_bonferroni_correction()) is applied per statistic, treating the \(N (N - 1) / 2\) comparisons for that statistic as one family. Pass correction=None to disable this and only get the raw, uncorrected p-values.

Parameters:

samples (Tuple[NDArray[int64] | NDArray[float64] | PredSampleWrapper, ...]) – A tuple of \(N \geq 2\) mutually paired samples to compare.
statistics (Mapping[str, Callable]) – A dictionary of statistics to compute on each sample.
groups (NDArray[int64] | None) – Groups indicating the subject for each measurement. Defaults to None.
labels (Tuple[str, ...] | None) – Labels for the samples, used to name each comparison as "{label_i} / {label_j}". Defaults to ("0", "1", ..., str(N-1)).
bootstrap_results (Dict[str, NDArray[float64]] | None) – Precomputed output of bootstrap_arrays() for these exact samples (e.g. to reuse the same bootstrap draws for a corrected and an uncorrected call without resampling twice). If None (the default), it is computed internally via bootstrap_arrays().
alpha (float) – A significance level for confidence intervals (from 0 to 1). Defaults to 0.05.
n_bootstrap (int) – The number of bootstrap iterations. Defaults to 5000.
correction (str | None) – Multiple-comparison correction to apply to the p-values, or None to disable it. Only "holm" is currently supported. Defaults to "holm".
seed (int | None) – Random seed. Defaults to None.
silent (bool) – Whether to execute the function silently, i.e. not showing the progress bar. Defaults to False.

Returns:

A dictionary keyed by statistic tag, then by comparison label ("{label_i} / {label_j}"), holding a dict with keys p_value (raw, uncorrected), p_value_adjusted (the Holm-adjusted p-value, or None if correction is None), diff (the observed effect size), ci_es/ci_s1/ci_s2 (percentile confidence intervals for the effect size and for each of the two samples in the pair), and emp_s1/emp_s2 (the empirical statistic for each of the two samples in the pair).

Return type:

Dict[str, Dict[str, Dict[str, float]]]

stambo.bootstrap_arrays(arrays, statistics, groups=None, n_bootstrap=5000, seed=None, silent=False)[source]

Bootstraps a tuple of mutually paired samples (e.g. predictions from \(N \geq 2\) models evaluated on the same test set).

All samples are resampled with the same indices at every bootstrap iteration (a paired design), optionally resampling whole groups/clusters instead of individual rows when groups is provided (see two_sample_test() for the rationale). This is the building block used by pairwise_bootstrap_test(), but it is also useful on its own if you want the raw bootstrap distributions for \(N\) samples at once (e.g. to compute your own downstream statistic across all of them jointly).

Parameters:

arrays (Tuple[NDArray[int64] | NDArray[float64] | PredSampleWrapper, ...]) – A tuple of \(N \geq 2\) samples to bootstrap together. Every array must have the same length, and either all be numpy.ndarray (with the same dtype) or all be PredSampleWrapper (with the same multiclass, threshold, and groups).
statistics (Mapping[str, Callable]) – A dictionary of statistics to compute on each sample.
groups (NDArray[int64] | None) – Groups indicating the subject for each measurement. Defaults to None. Must not be passed if the samples are PredSampleWrapper objects that already carry groups.
n_bootstrap (int) – The number of bootstrap iterations. Defaults to 5000.
seed (int | None) – Random seed. Defaults to None.
silent (bool) – Whether to execute the function silently, i.e. not showing the progress bar. Defaults to False.

Returns:

A dictionary keyed by statistic tag, each holding an array of shape (n_bootstrap, N) with the bootstrap replicates of that statistic for every sample.

Return type:

Dict[str, NDArray[float64]]

stambo.holm_bonferroni_correction(p_values)[source]

Holm-Bonferroni step-down correction for multiple comparisons.

Given \(m\) p-values from a family of hypothesis tests, adjusts them so that the family-wise error rate (the probability of at least one false positive across the whole family) is controlled at the nominal level, while being less conservative than a plain Bonferroni correction (dividing every p-value by \(m\)).

Sort the p-values ascending, \(p_{(1)} \leq \dots \leq p_{(m)}\). The adjusted p-value for the \(k\)-th smallest (1-indexed) is

\[\tilde p_{(k)} = \max_{l \leq k} \min\left(1, (m - l + 1) \, p_{(l)}\right),\]

i.e. each p-value is multiplied by the number of remaining hypotheses at its rank, capped at 1, and then enforced to be non-decreasing (monotone) with rank.

Parameters:: p_values (Sequence[float] | NDArray[float64]) – A 1D sequence of raw p-values.
Returns:: An array of the same length and order as p_values, with the Holm-adjusted p-values.
Return type:: NDArray[float64]

stambo.to_latex(report, m1_name='M1', m2_name='M2', n_digits=2)[source]

Converts a report returned by StamBO into a LaTeX table for convenient viewing.

Note: The alternative hypothesis is that the second model (M2) is different from the first model (M1). The p-value is the two-tailed p-value.

Parameters:

report (Dict[str, NDArray[float64]]) – Dictionary with metrics in the StamBO-generated format.
m1_name (str) – Name to assign to the first model row. Defaults to M1.
m2_name (str) – Name to assign to the second model row. Defaults to M2.
n_digits (int) – Number of digits to round to. Defaults to 2.

Returns:

A cut-and-paste LaTeX table in the tabular environment.

Return type:

str

stambo.pairwise_to_latex(report, n_digits=2)[source]

Converts a report returned by stambo.pairwise_bootstrap_test() / stambo.compare_models_pairwise() into a LaTeX table, one row per pairwise comparison.

For each statistic, the table shows the observed effect size with its confidence interval, and the p-value. If the report was produced with a multiple-comparison correction (i.e. at least one comparison has a non-None p_value_adjusted), the adjusted p-value is shown in parentheses next to the raw one; otherwise only the raw p-value is shown.

Parameters:

report (Dict[str, Dict[str, Dict[str, Any]]]) – Dictionary in the format returned by pairwise_bootstrap_test/compare_models_pairwise: {statistic: {"label_i / label_j": {"p_value": ..., "p_value_adjusted": ..., "diff": ..., "ci_es": (lo, hi), ...}}}.
n_digits (int) – Number of digits to round to. Defaults to 2.

Returns:

A cut-and-paste LaTeX table in the tabular environment.

Return type:

str

stambo.to_dict(report)[source]

Converts a report returned by stambo.two_sample_test() / stambo.compare_models() (a dict of positional arrays) into a dict of named fields, using the same field names already used by stambo.pairwise_bootstrap_test() / stambo.compare_models_pairwise() (p_value, diff, ci_es, ci_s1, ci_s2, emp_s1, emp_s2) – one consistent vocabulary across the whole library, instead of remembering the positional array order (index 0 is the p-value, index 1 the effect size, etc.).

Every value is cast to a plain Python float (never numpy.float64), so the result is guaranteed to round-trip through json.dumps – unlike the raw report, whose numpy.ndarray values are not JSON-serializable.

Parameters:: report (Dict[str, NDArray[float64]]) – Dictionary in the format returned by two_sample_test/compare_models: {statistic: array([p_value, diff, ci_es_lo, ci_es_hi, emp_s1, ci_s1_lo, ci_s1_hi, emp_s2, ci_s2_lo, ci_s2_hi])}.
Returns:: {statistic: {"p_value": ..., "diff": ..., "ci_es": (lo, hi), "ci_s1": (lo, hi), "ci_s2": (lo, hi), "emp_s1": ..., "emp_s2": ...}}.
Return type:: Dict[str, Dict[str, float | Tuple[float, float]]]

class stambo.PredSampleWrapper(predictions, gt, groups=None, multiclass=True, threshold=0.5, cached_am=None)[source]

Bases: object

Wraps predictions and targets in one object.

Parameters:

predictions (NDArray[float64] | NDArray[int64]) – Model predictions to wrap.
gt (NDArray[float64] | NDArray[int64]) – Ground-truth labels.
groups (NDArray[int64] | None) – Optional groups indicating the subject for each measurement. When several PredSampleWrapper samples are compared together (e.g. via stambo.pairwise_bootstrap_test()), all of them must carry the same groups. Defaults to None.
multiclass (bool) – Whether the predictions correspond to a multiclass classifier. Defaults to True.
threshold (float | None) – Threshold to apply to binary predictions when multiclass is False. Defaults to 0.5.
cached_am (NDArray[int64] | None) – Optional cached argmax / thresholded predictions to reuse.

__getitem__(idx)[source]

Give access to the predictions and the ground truth by index or a set of indices.

Parameters:: idx (int | NDArray[int64]) – Single index or collection of indices.
Returns:: Either a tuple containing the predictions, argmaxed predictions, and ground truth for a single index, or a new PredSampleWrapper restricted to the provided indices.
Return type:: Tuple[float, int, float | int] | TypeAliasForwardRef(‘stambo.PredSampleWrapper’)