Contextual bandits

A small action set, fast feedback, and a cost to showing the wrong thing. If feedback is slow or actions have long consequences, this is the budget-constrained optimisation playbook.

Logged context, action, and reward, plus the probability the action was shown. Without that propensity you cannot evaluate offline.

Context features that are known at decision time. Nothing that arrives after the impression.

Temporal. Offline replay on logged data.

A fixed A/B split is the baseline. Epsilon-greedy, then Thompson sampling, then LinUCB or a contextual policy when context matters.

Updates are online; the hyperparameter is the exploration rate and how fast it decays.

Replay and inverse propensity scoring on logs. Doubly robust when the logging policy was skewed.

The bandit against the fixed split, with guardrails on the arms it might starve.

Log propensities forever. Watch for the loop where the bandit stops exploring and the data stops informing.