Outlier Detection

Purpose

Outliers in historical demand data corrupt every downstream algorithm: characterization flags are skewed, model parameters are mis-estimated, and backtesting metrics are inflated. ForecastAI's outlier detection module (files/outlier/detection.py, class OutlierDetector) runs immediately after ETL and before characterization.

The module:

1. Detects which observations are outliers using one of three statistical methods.
2. Corrects or flags the outlier values using one of four strategies.
3. Stores every detected outlier (original value, corrected value, z-score, bounds) in the detected_outliers table.
4. Writes the corrected value back to demand_actuals.corrected_qty.

flowchart LR
    A[demand_actuals.qty] --> B[OutlierDetector]
    B --> C{Detection method}
    C --> D[IQR]
    C --> E[Z-Score / Modified Z-Score]
    C --> F[STL Residuals]
    D & E & F --> G[is_outlier mask\nlower_bound, upper_bound]
    G --> H{Correction method}
    H --> I[clip]
    H --> J[interpolate]
    H --> K[median]
    H --> L[none - flag only]
    I & J & K & L --> M[corrected_qty written\nto demand_actuals]
    G --> N[detected_outliers table\norig value, z_score, bounds]

Minimum series length: Series with fewer than min_observations (default: 6) points are skipped — too short for reliable statistical bounds.

Figure: Detection-method decision flow — STL residuals require at least two seasonal cycles, otherwise fall back to IQR.

flowchart TD
    S["demand_actuals.qty"] --> G{"n < min_observations<br/>(default 6)?"}
    G -- yes --> SK["Skip series"]
    G -- no --> M{"detection_method"}
    M -- iqr --> IQR["IQR<br/>Q1 - 1.5*IQR, Q3 + 1.5*IQR"]
    M -- zscore --> ZS["Modified Z-score<br/>median + MAD, |z| > 3.0"]
    M -- stl_residuals --> CK{"n >= 2 * seasonal_period?"}
    CK -- no --> IQR
    CK -- yes --> STL["STL robust decomposition<br/>z-score on residuals"]
    IQR & ZS & STL --> MS["is_outlier mask<br/>lower_bound, upper_bound"]

---

Method 1: IQR (Interquartile Range)

The IQR method is the default. It is non-parametric and robust: it does not assume a specific distribution, and is not sensitive to extreme values (the very outliers it is designed to find).

Algorithm (_detect_iqr):

1. Compute the 25th and 75th percentiles: \(Q_1\) and \(Q_3\)
2. Compute the IQR: \(IQR = Q_3 - Q_1\)
3. Compute bounds:

\[\text{lower} = Q_1 - k \cdot IQR\]

\[\text{upper} = Q_3 + k \cdot IQR\]

1. Flag as outlier: any observation where \(y < \text{lower}\) or \(y > \text{upper}\)

Multiplier \(k\): Default is 1.5 (the classic Tukey fence). A smaller \(k\) (e.g., 1.0) is more aggressive (more outliers flagged). A larger \(k\) (e.g., 3.0) is more conservative.

\[\text{is\_outlier}_t = \begin{cases} \text{True} & \text{if } y_t < Q_1 - 1.5 \cdot IQR \text{ or } y_t > Q_3 + 1.5 \cdot IQR \\ \text{False} & \text{otherwise} \end{cases}\]

Supply chain context: IQR works well for most demand series. It is symmetric, which can occasionally over-flag high-demand spikes on right-skewed distributions (e.g., promotional orders). In those cases, consider using a larger multiplier or the STL residuals method.

Configuration:

outlier_detection:
  detection_method: iqr
  iqr:
    multiplier: 1.5

---

Method 2: Modified Z-Score

The standard Z-score uses the mean and standard deviation, which are themselves sensitive to outliers. The modified Z-score uses the median and Median Absolute Deviation (MAD) instead, making it robust.

Algorithm (_detect_zscore with use_mad: true, the default):

1. Compute median: \(\tilde{y} = \text{median}(y)\)
2. Compute MAD: \(\text{MAD} = \text{median}(|y_t - \tilde{y}|)\)

3. Scale MAD for consistency with a normal distribution: \(\hat{\sigma} = 1.4826 \cdot \text{MAD}\)

   The factor 1.4826 comes from the normal distribution: \(\text{MAD}/\sigma \approx 0.6745\) for a normal distribution, so \(\sigma \approx 1.4826 \cdot \text{MAD}\).

4. Compute modified z-scores:

\[z_t = \frac{y_t - \tilde{y}}{\hat{\sigma}}\]

1. Flag as outlier when \(|z_t| > \text{threshold}\) (default 3.0):

\[\text{is\_outlier}_t = |z_t| > 3.0\]

Bounds (for correction and reporting):

\[\text{lower} = \tilde{y} - 3.0 \cdot \hat{\sigma}\]

\[\text{upper} = \tilde{y} + 3.0 \cdot \hat{\sigma}\]

Fallback to mean/std: When use_mad: false, the standard Z-score is used:

\[z_t = \frac{y_t - \bar{y}}{\sigma}\]

where \(\bar{y}\) is the mean and \(\sigma\) is the standard deviation. This is less robust but may be preferred for near-normal distributions.

Edge case: If \(\hat{\sigma} = 0\) (all values identical), no outliers are returned.

Configuration:

outlier_detection:
  detection_method: zscore
  zscore:
    threshold: 3.0
    use_mad: true    # false = standard mean/std z-score

---

Method 3: STL Residuals

Seasonal-Trend Decomposition using LOESS (STL) separates a series into trend, seasonal, and remainder components. The remainder (residuals) should be white noise; large residuals indicate anomalies that cannot be explained by the regular trend or seasonal pattern.

This method is the most sophisticated and most appropriate for series with strong seasonality, where the IQR/Z-score methods would incorrectly flag seasonal peaks as outliers.

Algorithm (_detect_stl_residuals):

Pre-check: Series must have at least \(2 \times \text{seasonal\_period}\) observations. If not, falls back to IQR.

1. Interpolate any NaN values linearly before STL fitting (STL requires no missing values).

2. Fit robust STL decomposition: STL(series, period=seasonal_period, robust=True).fit()

   The robust=True flag uses iteratively reweighted least squares in the LOESS smoother, reducing the influence of outliers on the trend and seasonal component estimates.

3. Extract residuals \(r_t\) from the fit: result.resid

4. Apply modified Z-score to residuals:

\[\tilde{r} = \text{median}(r), \quad \text{MAD}_r = \text{median}(|r_t - \tilde{r}|)\]

\[z_t^{(r)} = \frac{|r_t - \tilde{r}|}{1.4826 \cdot \text{MAD}_r}\]

1. Flag as outlier when \(z_t^{(r)} > \text{residual\_threshold}\) (default 3.0):

\[\text{is\_outlier}_t = z_t^{(r)} > 3.0\]

Bounds (approximate, in original scale): 1st and 99th percentiles of the original series values (used for correction reference only — the actual outlier decision is made on residuals).

STL decomposition model:

\[y_t = T_t + S_t + R_t\]

where \(T_t\) is the trend (LOESS smoother), \(S_t\) is the seasonal component, and \(R_t\) is the remainder (residual). An observation is flagged only when its \(R_t\) is anomalously large — meaning the value cannot be explained by either the trend or the seasonal pattern.

Configuration:

outlier_detection:
  detection_method: stl_residuals
  stl_residuals:
    seasonal_period: 12     # periods in one cycle (12 for monthly, 52 for weekly)
    residual_threshold: 3.0

When to use STL

Use STL residuals for series with strong, stable seasonality (e.g., consumer goods with holiday peaks). IQR would incorrectly flag the peaks as outliers. STL correctly identifies that the peaks are expected seasonal behaviour and only flags values that deviate from the expected seasonal level.

---

Correction Methods

Once outliers are flagged, one of four correction strategies is applied.

Figure: Correction pipeline — the chosen strategy writes corrected_qty to demand_actuals (qty unchanged) while every detected outlier is also logged to detected_outliers.

flowchart TD
    MS["is_outlier mask<br/>+ bounds"] --> CM{"correction_method"}
    CM -- clip --> CL["y_corr = min(max(y, lower), upper)"]
    CM -- interpolate --> IP["linear interpolation from neighbors<br/>bfill / ffill at edges"]
    CM -- median --> MD["rolling-window median<br/>window=5, center"]
    CM -- none --> NO["y_corr = y (flag only)"]
    CL & IP & MD & NO --> DA["demand_actuals.corrected_qty<br/>(qty column unchanged)"]
    MS --> DO["detected_outliers table<br/>original, corrected, z_score, bounds"]

Clip (default)

Cap the outlier value at the bound:

\[y_t^{(\text{corr})} = \begin{cases} \text{lower} & \text{if } y_t < \text{lower and is\_outlier} \\ \text{upper} & \text{if } y_t > \text{upper and is\_outlier} \\ y_t & \text{otherwise} \end{cases}\]

When to use: Preserves the information that an extreme event occurred (the corrected value is still higher/lower than normal), but removes the distorting magnitude. Good for promotional demand spikes.

Interpolate

Replace the outlier value with a linear interpolation from its neighbours:

\[y_t^{(\text{corr})} = y_{t-1} + \frac{(t - (t-1))}{(t+1) - (t-1)} \cdot (y_{t+1} - y_{t-1})\]

Edges (first or last observation are outliers) are filled with backward-fill then forward-fill. The interpolation method is configurable (interpolation_method: default 'linear').

When to use: Best for data-entry errors or isolated erroneous spikes that should genuinely be replaced.

Median

Replace the outlier value with the rolling-window median:

\[y_t^{(\text{corr})} = \text{median}(y_{t-w/2}, \ldots, y_{t+w/2})\]

where \(w\) is the window size (default: 5, center=True, min_periods=1).

When to use: More stable than interpolation when the series has high local variability. Preserves the general level of demand around the outlier.

None (flag only)

No correction is applied. The original values are left unchanged. The detected_outliers table is still populated for review, but corrected_qty equals qty.

When to use: When you want to review outliers before deciding on a correction, or when the business process requires human approval of corrections.

Configuration:

outlier_detection:
  correction_method: clip    # clip / interpolate / median / none
  correction:
    interpolation_method: linear
    median_window: 5

---

Z-Score Reporting

Regardless of which detection method is configured, the _compute_zscores function always computes a modified z-score for every observation (using median/MAD). This score is stored in the detected_outliers table alongside the detection result, providing a comparable severity measure across different detection methods.

\[z_t^{(\text{report})} = \frac{y_t - \tilde{y}}{1.4826 \cdot \text{MAD}}\]

---

Database Storage

detected_outliers table — one row per detected outlier:

Column	Description
unique_id	Series identifier
date	Date of the outlier observation
original_value	Original demand quantity
corrected_value	Corrected demand quantity (same as original if correction_method = 'none')
detection_method	Which method flagged it (iqr, zscore, stl_residuals)
correction_method	How it was corrected
z_score	Modified z-score of the original value
lower_bound	Lower detection threshold
upper_bound	Upper detection threshold

demand_actuals.corrected_qty — the corrected demand value used by all downstream steps (characterization, forecasting, backtesting). The original qty column is never modified.

---

Full Configuration Reference

outlier_detection:
  enabled: true
  detection_method: iqr          # iqr / zscore / stl_residuals
  correction_method: clip        # clip / interpolate / median / none
  min_observations: 6            # skip series shorter than this

  iqr:
    multiplier: 1.5              # Tukey fence multiplier k

  zscore:
    threshold: 3.0               # |z| threshold
    use_mad: true                # true = robust MAD; false = mean/std

  stl_residuals:
    seasonal_period: 12          # number of periods in one cycle
    residual_threshold: 3.0      # |z| threshold on residuals

  correction:
    interpolation_method: linear
    median_window: 5

---

Choosing a Method

Scenario	Recommended method
General-purpose, no strong seasonality	iqr (default)
Near-normal distribution, no seasonality	zscore
Strong seasonal pattern (e.g., annual peaks)	stl_residuals
Series where outliers should never be removed	none
Data entry errors (isolated spikes)	interpolate
Promotional demand (real but extreme)	clip

IQR and zero-heavy series

For intermittent demand series with many zeros, \(Q_1 = 0\), \(Q_3\) may be 0 or small, and the IQR can be near-zero. This can make the bounds very tight, flagging normal non-zero observations as outliers. Consider setting a larger iqr.multiplier (e.g., 3.0) or using zscore for intermittent series.