Backtesting & Method Selection¶

Purpose¶

Characterization routes a series to a pool of candidate methods. Backtesting evaluates each candidate empirically on held-out history, producing accuracy metrics for every method. Best method selection then uses a composite weighted score to pick the single winner per series.

This walk-forward empirical evaluation is what separates ForecastAI's method selection from fixed rules: the winning method is the one that has historically been most accurate for that specific series.

flowchart LR
    A[Candidate methods\nfrom characterization] --> B[Rolling Window\nBacktesting]
    B --> C[series_backtest_metrics\ntable]
    C --> D[Composite Weighted\nScoring]
    D --> E[series_best_methods\ntable]
    E --> F[Forward Forecast\nusing best method]

Rolling Window Walk-Forward Backtesting¶

What is walk-forward validation?¶

Walk-forward validation simulates the real forecasting situation: at each forecast origin, the model is trained on all data available up to that point and evaluated on the next \(W\) periods. This is the only valid way to evaluate time series models — using future data to train and past data to test (the "leakage" problem) produces misleadingly optimistic metrics.

Figure: Expanding-window walk-forward scheme (n=100, window_size=8, n_tests=4) — the training set grows with each successive origin.

gantt
    title Expanding-window walk-forward (n=100, window_size=8, n_tests=4)
    dateFormat X
    axisFormat %s
    section Fold 1
    train   :0, 40
    test    :40, 48
    section Fold 2
    train   :0, 53
    test    :53, 61
    section Fold 3
    train   :0, 66
    test    :66, 74
    section Fold 4
    train   :0, 79
    test    :79, 87

|--------- Training ---------|--- Test 1 ---|
                              ↑ Origin 1

|----------- Training -----------|--- Test 2 ---|
                                  ↑ Origin 2

|-------------- Training --------------|--- Test 3 ---|
                                        ↑ Origin 3

|---------------  Training -------------------|--- Test 4 ---|
                                               ↑ Origin 4
 0             40          53        66       79   87        100

At each origin, the model sees only past data. The test set covers the next window_size periods. The model forecasts window_size steps ahead. Actuals are compared to forecasts.

Algorithm¶

Configuration:

forecasting:
  backtesting:
    backtest_horizon: 60    # total periods reserved for backtesting
    window_size: 8          # forecast window per test (steps ahead)
    n_tests: 4              # number of forecast origins
    min_train_size: 24      # minimum training periods before first origin

Step-by-step (from files/evaluation/):

Given series of length \(n\):

Step 1 — Clamp:

\[\text{\_horizon} = \min(\text{backtest\_horizon},\; n - 1)\]

\[\text{\_window} = \min(\text{window\_size},\; \text{backtest\_horizon},\; \text{forecast\_horizon})\]

Step 2 — Origin boundaries:

\[\text{first\_origin} = \max(\text{min\_train\_size},\; n - \text{backtest\_horizon})\]

\[\text{last\_possible\_origin} = n - \text{window\_size}\]

\[\text{available\_range} = \text{last\_possible\_origin} - \text{first\_origin}\]

Step 3 — Step size between origins:

If n_tests = 1: single origin at first_origin
If n_tests <= 0 or exceeds max possible: test at every valid position (step = 1)
Otherwise: step = max(1, available_range / (n_tests - 1))

Step 4 — Generate origins:

\[\text{origins} = [\text{first\_origin},\; \text{first\_origin} + \text{step},\; \ldots]\]

Fallback: If available_range < 0, first_origin = max(1, n // 2).

At each origin:

\[\text{train} = y_{1:\text{origin}}\]

\[\text{test} = y_{\text{origin}:\text{origin}+\text{window\_size}}\]

The model is fit on train, forecasts window_size steps ahead, and the forecast is compared to test.

Expanding vs. rolling window¶

ForecastAI uses an expanding window: train = series[:origin_idx]. The training set grows with each successive origin. This maximizes the use of available data and is appropriate for most time series where more data improves model accuracy.

Which methods are backtested?¶

Method family	Backtested?	Reason
Statistical (AutoARIMA, AutoETS, AutoTheta, MSTL, Croston, etc.)	Yes	Fast enough for rolling re-training
ML (LightGBM, XGBoost)	Yes (default mode)	Included in all-methods mode
Neural (NHITS, NBEATS, PatchTST, TFT, DeepAR)	No	Too computationally expensive for rolling re-training
Foundation (TimesFM)	No	Zero-shot — no training loop

Neural and foundation models are selected for forward forecasting based on their design properties and the characterization-based routing, not backtesting.

Per-Origin Forecast Storage¶

Every individual forecast at every origin is stored for visualization:

-- stored in forecasts_by_origin:
(item_id, site_id, method, forecast_origin, horizon_step, point_forecast, actual_value)

This powers the "Forecasts by Origin" chart in the Time Series Viewer, showing all backtest windows overlaid on the actual series.

Per-Series Backtesting Overrides¶

Backtesting parameters can be overridden per-series via the hyperparameter_overrides table with method = '_backtesting':

{
  "backtest_horizon": 104,
  "window_size": 13,
  "n_tests": 6
}

This allows, for example, running a longer backtest for A-class items where precision is critical.

Accuracy Metrics¶

All metrics are computed in files/evaluation/ and stored in the series_backtest_metrics table.

Point Forecast Metrics¶

MAE (Mean Absolute Error):

\[MAE = \frac{1}{n} \sum_{t=1}^{n} |y_t - \hat{y}_t|\]

Scale-dependent. Interpretable in the original units (e.g., units of demand). Linear penalty — each unit of error adds equally. Good for supply chain: treating a 10-unit error and a 100-unit error differently (as RMSE does) can be misleading when demand is lumpy.

RMSE (Root Mean Square Error):

\[RMSE = \sqrt{\frac{1}{n} \sum_{t=1}^{n} (y_t - \hat{y}_t)^2}\]

Penalizes large errors quadratically — a 100-unit error contributes 100× more than a 10-unit error. More sensitive to outliers than MAE. Useful when large forecast errors have disproportionate business impact (e.g., stockouts are much more costly than overstock).

MAPE (Mean Absolute Percentage Error):

\[MAPE = \frac{100}{n} \sum_{t=1}^{n} \left|\frac{y_t - \hat{y}_t}{y_t}\right|\]

Percentage metric — scale-free comparison across series with different demand magnitudes. Undefined when \(y_t = 0\) (common in intermittent demand). Asymmetric: an over-forecast and under-forecast of the same absolute magnitude score differently.

sMAPE (Symmetric MAPE):

\[sMAPE = \frac{100}{n} \sum_{t=1}^{n} \frac{|y_t - \hat{y}_t|}{(|y_t| + |\hat{y}_t|) / 2}\]

Bounded in \([0, 200]\). Symmetric: over-forecasting and under-forecasting of the same absolute magnitude score the same. Still problematic when both \(y_t = 0\) and \(\hat{y}_t = 0\) (denominator = 0).

Bias:

\[Bias = \frac{1}{n} \sum_{t=1}^{n} (\hat{y}_t - y_t)\]

(In the composite scoring: \(Bias = \frac{\sum(\hat{y}_t - y_t)}{\sum y_t}\) — normalized by total demand)

Positive bias: systematic over-forecasting → excess inventory buildup
Negative bias: systematic under-forecasting → stockouts and service level failures
Zero bias: no systematic direction error (preferred)

Bias is arguably the most supply-chain-critical metric: a model that is on average accurate but consistently over-forecasts will cause inventory waste; one that consistently under-forecasts will cause stockouts.

MASE (Mean Absolute Scaled Error):

\[MASE = \frac{MAE}{\frac{1}{n-1} \sum_{t=2}^{n} |y_t - y_{t-1}|}\]

The denominator is the MAE of a naïve one-step forecast (just use the previous value). MASE > 1 means the method is worse than naïve. MASE < 1 means it is better. Scale-free and meaningful even for zero-demand periods.

WAPE (Weighted Absolute Percentage Error) — also called weighted MAPE:

\[WAPE = \frac{\sum_{t=1}^{n} |y_t - \hat{y}_t|}{\sum_{t=1}^{n} y_t}\]

Equivalent to weighting errors by the demand volume. Avoids division by zero since the denominator is total demand (positive). More robust than MAPE for intermittent series.

Probabilistic Metrics¶

Coverage at \(\alpha\)%: Fraction of actual observations that fall within the \(\alpha\)% prediction interval:

\[\text{Coverage}_\alpha = \frac{1}{n} \sum_{t=1}^{n} \mathbf{1}\left[\hat{y}_t^{(L_\alpha)} \leq y_t \leq \hat{y}_t^{(U_\alpha)}\right]\]

For 90% coverage: \(L_{90} = Q_{0.05}\), \(U_{90} = Q_{0.95}\).

A well-calibrated model should have coverage_90 ≈ 0.9. Systematic undercoverage (coverage_90 < 0.9) means intervals are too narrow — stockouts. Overcoverage means intervals are too wide — excess inventory.

Winkler Score (for 90% interval at significance level \(\alpha = 0.10\)):

\[WS = \frac{1}{n} \sum_{t=1}^{n} \left(U_t - L_t\right) + \frac{2}{\alpha} \left[\max(L_t - y_t, 0) + \max(y_t - U_t, 0)\right]\]

The first term penalizes wide intervals (lower = tighter = better). The second term penalizes boundary violations with a penalty proportional to the miss magnitude. Lower is better.

CRPS (Continuous Ranked Probability Score):

\[CRPS(\hat{F}, y) = \int_{-\infty}^{\infty} \left(\hat{F}(z) - \mathbf{1}(z \geq y)\right)^2 dz\]

where \(\hat{F}\) is the predicted CDF. Implemented via trapezoidal integration over the quantile pairs. CRPS generalizes MAE to full predictive distributions: when a point forecast is used, CRPS = MAE.

Lower CRPS = better calibrated probabilistic forecast.

Quantile Loss (Pinball Loss, averaged over all quantile levels \(Q\)):

\[QL = \frac{1}{|Q| \cdot n} \sum_{q \in Q} \sum_{t=1}^{n} \max\left(q(y_t - \hat{y}_t^{(q)}),\; (q-1)(y_t - \hat{y}_t^{(q)})\right)\]

Lower = better. This is the training objective for neural models (MQLoss) and is also computed during evaluation for consistency.

Information Criteria¶

Computed from in-sample residuals (statistical models only), assuming Gaussian residuals:

\[\hat{\sigma}^2 = \frac{1}{n} \sum_{t=1}^{n} (y_t - \hat{y}_t^{\text{in-sample}})^2\]

\[\log \hat{L} = -\frac{n}{2} \left(\log(2\pi\hat{\sigma}^2) + 1\right)\]

\[AIC = -2\log\hat{L} + 2k\]

\[BIC = -2\log\hat{L} + k\log(n)\]

\[AICc = AIC + \frac{2k(k+1)}{n - k - 1}\]

where \(k\) is the number of estimated model parameters.

Lower is better for model comparison within the same series. AICc corrects AIC for small sample sizes.

Composite Weighted Scoring¶

After backtesting, all candidate methods for a series are ranked by a composite score that combines multiple metrics. Implementation: files/selection/best_method.py, class MethodSelector.

Default Weights¶

_DEFAULT_WEIGHTS = {
    "mae":        0.40,
    "rmse":       0.20,
    "bias":       0.15,
    "coverage_90": 0.15,
    "mase":       0.10,
}

These weights are configurable in scenario.parameters under backtesting.weights (or the legacy path best_method.weights).

Why a Composite Score?¶

No single metric captures all aspects of forecast quality for supply chain:

Metric alone	Problem
MAE only	Ignores bias direction — a consistently over-forecasting method could win
RMSE only	Sensitive to outliers — a method with one bad forecast loses to a method with systematic small errors
Bias only	Ignores accuracy magnitude — a perfectly unbiased but very inaccurate method could win
Coverage_90 only	Ignores point accuracy — extremely wide intervals achieve perfect coverage trivially
MASE only	Naïve benchmark may be weak on trending series, inflating scores

The composite balances: - 40% MAE: Primary accuracy metric, robust and interpretable - 20% RMSE: Extra weight on large errors (protection against worst-case weeks) - 15% Bias: Service level impact — systematic over/under-forecasting - 15% Coverage deviation: Interval calibration — do the stated uncertainty bands hold? - 10% MASE: Ensures the method beats the naïve benchmark

Algorithm (`_rank_methods_for_series`)¶

Figure: Error-metric aggregation pipeline — per-origin metrics collapse into a single normalized composite score per method.

flowchart TD
    R["series_backtest_metrics<br/>rows per (method, origin)"] --> S1["Step 1: mean metric per method<br/>across all origins"]
    S1 --> S2["Step 2: bias → |bias|<br/>coverage_90 → |coverage_90 - 0.90|"]
    S2 --> S3["Step 3: min-max normalize [0, 1]<br/>per metric across methods"]
    S3 --> S4["Step 4: weighted composite<br/>sum(w_c * m_c) / sum(w_c non-NaN)"]
    S4 --> S5["Step 5: sort ascending<br/>lower = better"]
    S5 --> O["series_best_methods<br/>best_method + all_rankings"]

Step 1 — Average across forecast origins:

For each method, compute the mean of each metric across all backtest windows:

\[\bar{m}_{\text{method}} = \frac{1}{N_{\text{origins}}} \sum_{\text{origin}} m_{\text{method, origin}}\]

Step 2 — Transform bias and coverage:

Bias uses absolute value (both over- and under-forecasting are penalized equally):

\[\text{bias} \leftarrow |\text{bias}|\]

Coverage_90 is transformed to measure deviation from the nominal 90% target:

\[\text{coverage}_{90} \leftarrow |\text{coverage}_{90} - 0.90|\]

Step 3 — Min-max normalization to [0, 1] (per metric, across all methods for this series):

\[\tilde{m}_{\text{method}} = \frac{\bar{m}_{\text{method}} - \min_j \bar{m}_j}{\max_j \bar{m}_j - \min_j \bar{m}_j}\]

If all methods share the same value: \(\tilde{m} = 0.0\) (no discriminating power for this metric).

Step 4 — Weighted composite score:

\[\text{composite}_{\text{method}} = \frac{\sum_{c \in \text{metrics}} w_c \cdot \tilde{m}_{c,\text{method}}}{\sum_{c \in \text{metrics}: \tilde{m}_{c} \text{ not NaN}} w_c}\]

The denominator adjusts for methods that are missing some metrics (e.g., intermittent models without coverage data, or ML models with internal validation only). This ensures a method missing coverage_90 is not penalized relative to one with coverage — its score is normalized by the sum of the weights it does have.

Step 5 — Rank:

Sort by composite_score ascending. Lower score = better.

Step 6 — Output:

Field	Description
`best_method`	Winning method name
`best_score`	Composite score of the winner
`runner_up_method`	Second-best method
`runner_up_score`	Composite score of runner-up
`all_rankings`	Dict of `{method: score}` sorted ascending

Worked Example¶

Three methods (AutoETS, AutoARIMA, MSTL) for one series, averaged over 4 backtest origins:

Method	MAE	RMSE	Bias	Coverage_90	MASE
AutoETS	120	180	5	0.88	0.92
AutoARIMA	140	210	-8	0.91	1.05
MSTL	115	175	12	0.85	0.88

After transformation (absolute bias, coverage deviation):

Method	MAE	RMSE	abs(Bias)	abs(Coverage-0.9)	MASE
AutoETS	120	180	5	0.02	0.92
AutoARIMA	140	210	8	0.01	1.05
MSTL	115	175	12	0.05	0.88

After min-max normalization (min=0, max=1 per column):

Method	MAE	RMSE	abs(Bias)	abs(Coverage-0.9)	MASE
AutoETS	0.20	0.14	0.00	0.25	0.24
AutoARIMA	1.00	1.00	0.43	0.00	1.00
MSTL	0.00	0.00	1.00	1.00	0.00

Composite scores (weights: 0.40, 0.20, 0.15, 0.15, 0.10):

\[\text{AutoETS} = 0.40(0.20) + 0.20(0.14) + 0.15(0.00) + 0.15(0.25) + 0.10(0.24) = 0.149\]

\[\text{AutoARIMA} = 0.40(1.00) + 0.20(1.00) + 0.15(0.43) + 0.15(0.00) + 0.10(1.00) = 0.764\]

\[\text{MSTL} = 0.40(0.00) + 0.20(0.00) + 0.15(1.00) + 0.15(1.00) + 0.10(0.00) = 0.300\]

Ranking: AutoETS (0.149) → winner; MSTL (0.300) → runner-up; AutoARIMA (0.764) → last.

Note that MSTL had the best MAE and RMSE, but its large bias and poor coverage hurt its composite score. AutoETS had moderate accuracy but excellent bias control and better coverage calibration.

ML Internal Validation Fallback¶

When rolling-window backtesting produces no metrics for ML methods (common when the series is too short for multiple rolling windows), the system activates a fallback:

Run the ML forecaster on the full series
Extract internal 80/20 train/val split metrics
Tag the resulting metrics with metric_source = 'internal_validation'
Append to the backtest metrics table

This ensures ML methods are never left without metrics for composite scoring comparison.

Database Tables¶

`series_backtest_metrics`¶

One row per (item_id, site_id, method, forecast_origin):

Column	Description
`pipeline_id`	Pipeline identifier
`scenario_id`	Scenario identifier
`item_id`	Item identifier
`site_id`	Site identifier
`item_name`	Built-in item name (CH mirror)
`site_name`	Built-in site name (CH mirror)
`method`	Forecasting method name
`forecast_origin`	Index of the forecast origin
`mae`	Mean Absolute Error
`rmse`	Root Mean Square Error
`mape`	Mean Absolute Percentage Error
`smape`	Symmetric MAPE
`bias`	Directional bias
`mase`	Mean Absolute Scaled Error
`coverage_50`	50% interval coverage
`coverage_80`	80% interval coverage
`coverage_90`	90% interval coverage
`coverage_95`	95% interval coverage
`winkler_score`	Winkler score (90% interval)
`crps`	Continuous Ranked Probability Score
`quantile_loss`	Average pinball loss
`aic`	Akaike Information Criterion
`bic`	Bayesian Information Criterion
`aicc`	Corrected AIC
`metric_source`	`'backtesting'` or `'internal_validation'`

`series_best_methods` (alias: `best_method_per_series`)¶

One row per (item_id, site_id, pipeline_id, scenario_id):

Column	Description
`item_id`	Item identifier
`site_id`	Site identifier
`pipeline_id`	Pipeline identifier
`scenario_id`	Scenario identifier
`best_method`	Winning method name
`best_score`	Composite score of the winner
`runner_up_method`	Second-best method name
`runner_up_score`	Composite score of the runner-up
`all_rankings`	JSON object: all methods with their scores

`forecasts_by_origin`¶

One row per (item_id, site_id, method, forecast_origin, horizon_step):

Column	Description
`item_id`	Item identifier
`site_id`	Site identifier
`method`	Method name
`forecast_origin`	Index of the forecast origin
`horizon_step`	Step ahead (1 to window_size)
`point_forecast`	Forecasted value
`actual_value`	Observed actual value

Incremental Backtest Cache — `PIPE_backtest_fingerprints`¶

Unchanged series are skipped across runs by storing a fingerprint per (pipeline_id, scenario_id, item_id, site_id) (ReplacingMergeTree(computed_at), read with FINAL, no unique_id column). The fingerprint is:

MD5(data_hash + sorted methods + overrides + param_hash)

where param_hash folds in every forecast parameter surface (forecast set, hyperparameters, backtesting/method-selection set, scenario-level overrides) — so a parameter-set or scenario-override change forces a re-backtest. The fingerprint payload key defaults_v was bumped from 2 to 3 when param_hash was added. A forced full rerun inserts a row with fingerprint = ''.

All metrics NaN

If every metric is NaN for all methods (e.g., entire test period has zero actuals making MAPE undefined), the _default_result fallback returns the first method in the candidate list with best_score = NaN. This is rare but can happen for highly intermittent series.

Methods missing some metrics

Intermittent models (CrostonOptimized, ADIDA, IMAPA) do not produce coverage metrics without conformal intervals (requires \(n \geq 2h+1\)). Their composite scores are normalized by the sum of available weights only (MAE + RMSE + Bias + MASE = 0.85 of the total weight). This is fair: they are not penalized for the missing 0.15.

Single method in the candidate pool

When only one method is available (after data sufficiency filtering), no ranking is needed — it wins by default with best_score = 0.0 (only one method means normalized score is always 0 after min-max).

Customizing weights for your business

Supply chains with very high stockout costs (e.g., pharmaceutical, aerospace MRO) should increase the bias weight — detecting systematic under-forecasting early is critical. Supply chains with high inventory carrying costs (e.g., perishables, fashion) should increase the mae weight relative to RMSE — consistent small accuracy matters more than avoiding occasional large misses.

# High-service-level configuration (minimize stockouts):
backtesting:
  weights:
    mae: 0.30
    rmse: 0.20
    bias: 0.30     # increased: bias matters most
    coverage_90: 0.15
    mase: 0.05

# High inventory cost configuration (minimize over-forecast):
backtesting:
  weights:
    mae: 0.45     # increased: raw accuracy
    rmse: 0.15
    bias: 0.25    # still important
    coverage_90: 0.10
    mase: 0.05

Visual Output in the Frontend¶

The Time Series Viewer (/series/:uniqueId) displays backtesting results in several ways:

Method comparison table: All backtested methods with their metric columns. The winning method is highlighted in green (#059669 / bg-emerald-*).
Forecasts by Origin chart: Plotly chart overlaying all backtest windows on the actual series. Each origin's forecast is shown as a colored line segment.
Ridge chart: Distribution of forecast errors per method across origins, useful for assessing consistency.
Best method badge: Shown prominently on the series detail card.

The API endpoint GET /api/series/:uniqueId/backtest-metrics returns all metrics for the series. GET /api/series/:uniqueId/forecasts-by-origin returns per-origin forecast/actual pairs.