Skip to content

Replay Engine

Table of Contents

Overview

The Replay Engine re-runs the supply + allocation planning pipeline iteratively over a configurable historical horizon. At each iteration, the engine advances the "planning date" by one step (daily or weekly), receives supply orders that have arrived by that date into on-hand inventory, and runs the full supply and allocation engines again.

This allows you to answer questions like:

  • What would our fill rate have been if we had used this planning configuration over the past year?
  • How would stock have evolved week by week under the current policy?
  • Where did backorders accumulate, and when did they clear?

The replay uses the same engines as the live planning process — not a simplified simulation — so the results accurately reflect what the system would have produced at each point in time.

Where to find it

The Replay screen is accessible from the 🔄 Replay link in the sidebar's Action group.

Core Concepts

Replay Run

A replay run defines the scope of a replay:

Parameter Description
Source pipeline The pipeline whose configuration (supply scenario, allocation parameters, forecast) will be replayed
Start date The first planning date — the beginning of the historical horizon
End date The last planning date — the end of the historical horizon
Granularity Step size: weekly (default) or daily

Each replay run creates its own replay pipeline (a copy of the source pipeline's configuration). The source pipeline's data is never modified.

Iteration

Each iteration is one complete planning cycle at a specific planning date:

  1. Receive supply: Firm supply orders that arrived since the previous iteration are "received" — their quantity is added to on-hand inventory (delta receive, so each order is received exactly once)
  2. Run supply planning: The full supply engine runs, generating new replenishment orders
  3. Run allocation: The allocation engine distributes available supply across demands
  4. Record metrics: Fill rate, total stock, and shortage are captured

Each iteration writes its output with archive_date = Monday of the planning week (the project-wide archive convention), so results are naturally historized and queryable across iterations.

Weekly granularity is Monday-aligned

Weekly iterations always fall on Mondays. The supply and allocation engines stamp every PIPE_* row with the Monday of the planning week, so the engine aligns iteration dates to the same convention — a run started mid-week begins on the Monday of that week.

Iteration dates and on-hand inventory

  • Orders that arrived before the replay's first iteration are assumed to already be reflected in the on-hand snapshot and are never re-received.
  • Orders arriving between two iterations are received into on-hand on the first iteration whose planning date is on/after the arrival — exactly once (each receipt is journaled).
  • Replays are side-effect-free: every on-hand delta applied during the run (receipts, and shortfall carries in Backorder mode) is journaled and reversed when the run ends — on success, failure and cancellation alike. The tenant's on-hand ends exactly where it started; the stock evolution lives in the per-iteration metrics. A crashed process leaves the journal behind, and the next run start (or the API's startup sweep) reverses it automatically.
  • Lost sale vs backorder: demand_carry_mode in the run config controls unfilled demand. With lost_sale (default) unallocated demand disappears; with backorder the shortfall is subtracted from on-hand so the next iteration sees the backlog.

Stock Evolution

Between iterations, the on-hand inventory evolves realistically:

  • Supply orders from previous iterations that arrive by the current planning date are received into stock
  • Demand that was allocated (filled) reduces available inventory
  • Unfulfilled demand (backorders) carries forward

Creating a Replay Run

  1. Navigate to the Replay screen (🔄 in the sidebar)
  2. Click + New Replay
  3. Fill in the form:
Field Required Description
Name Yes A descriptive name for this replay run
Description No Optional details
Source Pipeline Yes The pipeline to replay
Start Date Yes First iteration date (e.g. 2025-06-01)
End Date Yes Last iteration date (e.g. 2026-06-01)
Granularity Yes Weekly (default) or Daily
Demand Carry Mode Yes lost_sale (default) — unfulfilled demand disappears; backorder — shortfall carries into the next iteration's on-hand
  1. Click Create

New Replay Run form

The system creates a new replay pipeline (marked with is_replay = TRUE) that clones the source pipeline's supply and allocation scenario configuration. The source pipeline is not modified.

Running a Replay

After creating a replay run, its status is pending. To start it:

  1. Click the Start button in the runs table
  2. The status changes to running and the progress bar begins advancing

The replay runs in the background — you can navigate away and return later. The progress bar shows the current iteration out of the total.

Runtime estimate

Each iteration takes approximately 1 minute (supply + allocation). A 52-week replay runs in about 1 hour. A 365-day daily replay takes approximately 6 hours.

Replay Dashboard

Replay Engine dashboard with the list of replay runs

When you select a running or completed replay run, the dashboard shows a tab bar with three views:

  • Overview — the original panels: run config badges, Stock Over Time chart, Fill Rate & Shortfall chart, and the Iteration Details table
  • SKU Metrics — a sortable, searchable per-(item, site) grid of target vs. achieved fill rate and stock start/end
  • Worst Offenders — the SKUs that increased stock over the replay and the SKUs that missed their target fill rate

The panels of the Overview tab:

Stock Over Time

A line chart showing projected inventory across all iterations. Each line represents one item-site combination, with the X-axis showing planning dates and the Y-axis showing stock quantity.

Use this to identify:

  • Periods of low stock or stockouts
  • Inventory build-up patterns
  • Seasonal stock patterns under the replayed policy

Fill Rate & Shortfall Over Time

A combined chart with:

  • Fill rate % (left Y-axis, line chart): the percentage of demand that was allocated
  • Shortfall qty (right Y-axis, bar chart): the total unfulfilled quantity

Use this to identify:

  • Weeks where allocation was insufficient
  • Trends in service level
  • The impact of lead times on fill rate

Iteration Details Table

A row-by-row breakdown of every iteration:

Column Description
# Iteration number
Planning Date The date this iteration simulated
Status Completed, failed, or skipped
Fill Rate Percentage of demand allocated
Total Stock Sum of projected on-hand across all item-sites
Shortage Total shortage quantity
Duration Wall-clock time for this iteration

Stock Over Time

The stock evolution chart queries PIPE_supply_inventory_projection from ClickHouse, grouped by archive_date (one per iteration). Each series is an item-site combination.

The chart supports:

  • Hovering to see exact values
  • Zooming by clicking and dragging
  • Double-clicking to reset the zoom

Fill Rate Over Time

Fill rate is the overall fill rate of the allocation run — allocated quantity across all demands, divided by the total requested quantity:

allocated_total = SUM(allocated_qty) FROM PIPE_allocation_results        -- every allocated unit
requested_total = SUM(requested_qty) FROM PIPE_allocation_unfulfilled    -- shortfall demands
                 + (allocated_total - SUM(allocated_qty) FROM unfulfilled)  -- fully-filled demands
fill_rate       = allocated_total / requested_total

PIPE_allocation_unfulfilled only holds demands that ended short, so computing allocated/requested from that table alone would report the fill rate of the shortfall subset rather than of the whole run. Both tables are partitioned by (pipeline_id, archive_date), so each iteration's fill rate is computed independently.

Iteration Details

The iteration table reads from the replay_iteration PostgreSQL table. Each row contains a metrics JSONB column with the following structure:

{
  "total_stock": 15420.0,
  "total_shortage": 320.5,
  "avg_stock": 128.5,
  "total_allocated": 15100.0,
  "total_requested": 15420.0,
  "total_shortfall": 320.0,
  "fill_rate": 0.9792
}

SKU Metrics

The SKU Metrics tab shows one row per (item, site) that appears anywhere in the replay:

Column Description
Item / Site The item-site combination
Segment The binding segment (the one whose target applies)
Target % The segment's MEIO fill-rate target, from the source pipeline
Achieved % The fill rate achieved at the selected archive (iteration)
Diff (pp) Achieved − target, in percentage points (red when negative)
Stock Start Projected inventory at the first iteration (week 0)
Stock End Projected inventory at the last iteration (final week)
Δ Stock Stock End − Stock Start (red when stock grew)
Shortage Total shortfall quantity at the selected archive

Controls:

  • Archive selector — pick the iteration whose fill columns (Target/Achieved/Diff/Shortage) you want. Defaults to the last iteration. Stock Start/End are replay-level and never change with the selector.
  • Search — case-insensitive substring filter on item or site name.
  • Sort — click any column header to toggle ascending/descending. Default sort is Diff (pp) ascending (worst first); rows with no value always sort last.
  • Row click — opens the SKU drill-down detail for that (item, site) at the selected archive.

Where the target comes from

Replays do not re-run segmentation or MEIO, so the target fill rate is policy from the source pipeline: the SKU's latest segment membership → the highest MEIO group target across the SKU's segments (the binding group). SKUs without explicit membership use the default segment's target; SKUs with no resolvable target show — in the Target % column.

The achieved fill rate uses the project-standard overall formula across both allocation tables (see Fill Rate Over Time) — the unfulfilled table alone holds only the shortfall subset.

API: GET /api/replay/runs/{id}/sku-metrics?archive_date=YYYY-MM-DD — all rows are returned (client-side sort/search); archive_date defaults to the last iteration.

Worst Offenders

The Worst Offenders tab answers "what should I look at first?" with two fixed-order sections (the search box from SKU Metrics applies here too):

  • Stock increased (N) — every (item, site) whose stock grew over the replay (Δ Stock > 0), sorted by stock increase, largest first. Inventory that built up under the replayed policy is working capital to investigate.
  • Target missed (N) — every (item, site) with a target where the achieved fill rate fell short (Diff < 0), sorted by the miss, worst first.

Clicking a row opens the SKU drill-down detail.

SKU Drill-Down Detail

Clicking a row in either grid opens a full-screen detail modal for that (item, site):

  • Header — item name @ site name.
  • Archive navigation — ← Prev, an archive dropdown and Next → let you walk the replay iteration by iteration and watch the situation evolve (the buttons are disabled at the window's edges).
  • Inventory balance chart — projected inventory (line), demand (bars), shortage (red line) and safety stock (dashed) over the horizon's week dates: the demand that led to the supply, and the balance that led to backorders.
  • Three lists (side by side on wide screens):
    • Firm Demand — confirmed future orders from master.demand_actuals (is_forecast = FALSE) within 365 days of the archive date: date, qty, customer, priority
    • Recommended Orders — the replay engine's planned orders at that archive: type, release → arrival date, qty, status
    • Firm Supply Orders — the inbound order book (master_supply_orders_firm): type, arrival, qty, ERP status

API: GET /api/replay/runs/{id}/series-detail?item_id=&site_id=&archive_date= — archive_date defaults to the last iteration; the response carries prev_archive / next_archive for navigation.

Cancelling and Deleting

Cancelling a Running Replay

Click Cancel on a running replay. The engine checks for cancellation before each iteration, so the current iteration will complete before the replay stops. Partial results (iterations that already completed) are preserved.

Deleting a Replay Run

Click Delete on a completed, failed, or cancelled replay. This removes:

  • The replay run metadata
  • All iteration records
  • The replay pipeline

Deletion is permanent

Deleted replay data cannot be recovered. The source pipeline is never affected by deletion.

Limitations and Performance

Aspect Detail
One replay at a time Only one replay can run per source pipeline concurrently. Starting a second replay on the same pipeline will fail.
Forecast bootstrap The replay pipeline starts with empty PIPE_* data. On the first iteration the supply step auto-runs the full forecast chain (characterization → backtest → forecast → best-method) for the replay pipeline, then reuses it for every subsequent iteration. The first iteration therefore takes several minutes; later iterations run in seconds.
Engine runtime After the bootstrap, each iteration runs netting + supply + allocation (~10 s on the reference tenant). A 14-week replay ≈ 2–3 min plus one bootstrap.
Side-effect-free Every on-hand delta (receipts, backorder carries) is journaled and reversed when the run ends — success, failure or cancellation. A killed process leaves the journal, which the next run start (or the API startup sweep) reverses automatically. Concurrent on-hand edits made during the run are never clobbered.
ClickHouse partitions Each iteration writes a new archive bucket to PIPE_supply_orders, PIPE_supply_inventory_projection, PIPE_supply_capacity, PIPE_allocation_results and PIPE_allocation_unfulfilled. For daily granularity several iterations share one weekly (Monday) bucket — only the last state per bucket survives.
No demand simulation The replay replays real historical demand (demand actuals + the replay pipeline's own forecast). It does not simulate alternative demand scenarios.

Worked Example — theBicycle "Default" pipeline, 3-month weekly replay (Sep 2026)

The first end-to-end validation of the replay engine was run on the theBicycle tenant (replay run #2, replay pipeline 129):

Setting Value
Source pipeline Default (pipeline 4, prod)
Window 2026-06-01 → 2026-08-31 (14 weekly Monday iterations)
Demand carry mode lost_sale
Forecast bootstrap 580 series characterized + backtested on iteration 1 (~6 min)

Results

  • All 14 iterations completed; overall fill rate declined 70.1% → 57.1% as summer demand rose, total stock evolved 386k → 351k units, shortage grew 122k → 133k.
  • Exactly 6 firm supply orders (1040 units) were received into on-hand across the window — each once, on the iteration covering its arrival week (on_hand evolved 10 688 → 11 728 units during the run, +1 040 byte-exact with the order book). This run predates the side-effect-free journaling introduced later the same day: its receipts were kept. Current engine versions reverse all deltas at run end.
  • Every iteration wrote its own CH archive bucket for replay pipeline 129: 14 archive dates across PIPE_supply_orders (253k rows), PIPE_supply_inventory_projection (541k), PIPE_allocation_results (65k), PIPE_allocation_unfulfilled (26k) and PIPE_forecast_netting (331k).

How to inspect it

  1. UI: theBicycle tenant → Process Runner (/processes) → 🔄 Replay tab → click the "Default 3-month weekly replay" row. The dashboard shows the run list (completed, iteration 14/14) with three tabs: Overview (the Stock Over Time chart, the Fill Rate & Shortfall Over Time chart and the per-iteration table with metrics), SKU Metrics (743 item-site rows; at the last archive 204 rows missed their segment target) and Worst Offenders (at the last archive: 31 SKUs increased stock, 204 missed target — click any row for the per-iteration drill-down).
  2. API: GET /api/replay/runs/2/iterations (per-iteration metrics JSON), GET /api/replay/runs/2/stock-evolution, GET /api/replay/runs/2/fill-rate-evolution, GET /api/replay/runs/2/sku-metrics (per-(item, site) target vs. achieved fill + stock start/end), GET /api/replay/runs/2/series-detail?item_id=…&site_id=… (drill-down: timeline, netting, orders, firm demand, firm supply).
  3. ClickHouse (tenant thebicycle): every replay table is scoped by pipeline_id = 129; group by archive_date to walk the timeline, e.g.
SELECT archive_date,
       round(sum(shortage)) AS shortage,
       round(avg(projected_inventory), 1) AS avg_stock
FROM PIPE_supply_inventory_projection
WHERE pipeline_id = 129
GROUP BY archive_date
ORDER BY archive_date

Rolling back run #2's on-hand receipts (historical)

Run #2 was executed before the side-effect-free journaling landed, so its receipts were kept in the live on-hand (+1 040 units). A pre-replay snapshot was kept at master._replay_oh_manual_pre_replay (theBicycle). To restore:

TRUNCATE master.on_hand;
INSERT INTO master.on_hand SELECT * FROM master._replay_oh_manual_pre_replay;

Runs executed after the journaling change need no rollback — they reverse themselves.