Welcome to stambo!

About

Statistical Model Comparison with Bootstrap (STAMBO) focuses on statistically sound comparisons between models and samples by implementing the two-tailed bootstrap hypothesis tests:

We have abstracted the bootstrap two-sample test into a single function: stambo.two_sample_test(). To start using the library, one can simply compare just two means (the default assumpes paired design).

import stambo
...
seed = 42
res = stambo.two_sample_test(sample_1, sample_2, statistics={"Mean": lambda x: x.mean()})

If you would like to avoid the paired design, you can simply set the non_paired argument to True.

What makes this libarry different, is that we support implementation of bootsyrap across many metrics at the same time and clustered bootstrap. The latter is particularly useful when the data is from the same patient. Here is how we run it for the case when predictions come a dataset with repeated measurements from the same patient:

import stambo
...
seed = 42
results = stambo.compare_models(y_test, preds_1, preds_2, ("ROCAUC", "AP", "QKappa", "BACC", "MCC"), seed=seed, n_bootstrap=1000)
print(stambo.to_latex(results))

The above will print a LaTeX table, which one can easily copy-paste:

example table

If you have more than two models (or samples) to compare, stambo.compare_models_pairwise() (and its lower-level building block, stambo.pairwise_bootstrap_test()) run the bootstrap test on every pair, with a Holm-Bonferroni correction for the multiple comparisons applied by default:

import stambo
...
seed = 42
results = stambo.compare_models_pairwise(y_test, (preds_1, preds_2, preds_3), ("ROCAUC", "AP"), seed=seed, n_bootstrap=1000)
print(stambo.pairwise_to_latex(results))

See the Comparing more than two models with stambo example for a full walkthrough, including why the correction matters and how it interacts with clustered/grouped data.

Using stambo with AI coding agents

stambo is built to be used directly by AI coding agents (e.g. Claude Code, Codex) that write and run Python against the installed package. The repository ships an AGENTS.md file – a concise, verified cheat sheet covering which function to call, paired vs. non_paired vs. groups semantics, the two-tailed p-value convention, and the exact return-format schema – which coding agents pick up automatically as project context (a CLAUDE.md pointer is included too, for Claude Code’s own auto-load mechanism).

To make results easy for agents (and anyone scripting against stambo) to consume:

  • stambo.to_dict() converts a stambo.two_sample_test()/stambo.compare_models() report into the same named-field, JSON-serializable dict shape already used by the pairwise functions, instead of a positional numpy.ndarray.

  • The package ships a py.typed marker, so type checkers and IDE/agent tooling trust the type hints throughout the public API.

Documentation:

Examples: