Skip to content

Architecture Deep Dive

This page is a detailed technical companion to the system architecture overview. It covers the internal design decisions, component responsibilities, inter-process communication, and the data model that ties everything together.

For developer-level walkthroughs of individual components, see:

Design Principles

Mirabelle is built around five principles:

Principle How it manifests
Dual-store: PostgreSQL + ClickHouse Master data and pipeline configuration live in PostgreSQL. All pipeline outputs (forecasts, supply orders, projections, exceptions, netting) are written to ClickHouse PIPE_* tables for fast analytical queries. See ClickHouse section below.
Pipeline + Scenario scoping Every mutable output row is tagged with pipeline_id + scenario_id. This makes what-if analysis and multi-tenant isolation trivial.
Immutable master data item, site, demand_actuals, BOM, routes — never tagged with pipeline/scenario. They live in PostgreSQL only.
Single large API file files/api/main.py is intentionally one file. All endpoint behaviour is in one searchable place, shared state (caches, process registry) is straightforward.
Supply runs in a subprocess The supply planning step (run_pipeline.py --only supply) runs as a detached subprocess, not inside the API process. A DescendantTracker watchdog (files/utils/subprocess_watchdog.py) reaps any surviving worker processes after the step completes, preventing the API from being starved of threads.

Data Flow: Request Lifecycle

Every API request goes through this sequence:

sequenceDiagram
    participant Browser
    participant Vite as Vite Proxy :5173
    participant API as FastAPI :8002
    participant MW as JWTAuthMiddleware
    participant Cache as _account_cache
    participant DB as Tenant PostgreSQL

    Browser->>Vite: GET /api/series?pipeline_id=3
    Vite->>API: proxy forward (unchanged)
    API->>MW: intercept request
    MW->>MW: decode JWT, extract account_id + role
    MW->>Cache: lookup account_id
    alt Cache miss
        Cache->>DB: SELECT from master.accounts WHERE id=?
        DB-->>Cache: db_name, schema_name, connection_params
    end
    MW->>MW: set_account_context(conn_info)
    API->>DB: query scenario.forecast_results WHERE pipeline_id=3
    DB-->>API: rows
    API-->>Browser: JSON response

The tenant DB connection is resolved per-request from the JWT — this is what makes multi-tenancy transparent to all route handlers.

Pipeline Scoping: Immutable vs Mutable Data

The most important architectural concept in Mirabelle:

Type Store Tables Pipeline tag
Immutable PostgreSQL item, site, demand_actuals, bill_of_material, route, on_hand, causal_asset None — shared across all pipelines
Pipeline-scoped (config) PostgreSQL forecast_results_pg, meio_results, series_best_methods, exception_log, segment_membership pipeline_id + scenario_id
Pipeline-scoped (outputs) ClickHouse PIPE_forecast_results, PIPE_supply_orders, PIPE_supply_inventory_projection, PIPE_forecast_netting, PIPE_supply_exceptions, PIPE_supply_capacity, PIPE_meio_results, PIPE_series_characteristics pipeline_id + scenario_id

This design means:

  • Two pipelines share the same demand data while having independent forecasts and supply plans
  • What-if scenarios are free — create a new scenario_id, re-run MEIO with different targets, compare without touching production data
  • Rollback is free — point the UI to an older pipeline_id
  • Analytical queries against large result sets (supply projection, netting history) go to ClickHouse and are fast even for multi-year horizons

ClickHouse (PIPE_* tables)

All pipeline output tables use the naming convention PIPE_{step}_{entity} and live in the ClickHouse default database (one shared CH instance per server; tenant isolation is enforced by pipeline_id).

Table Populated by Key columns
PIPE_forecast_point_values Forecasting step pipeline_id, scenario_id, archive_date, item_id, site_id, method, forecast_week, point_forecast, q10/q50/q90
PIPE_forecast_fingerprints Forecasting step Pipeline-scoped forecast skip cache: combined input fingerprint MD5(data_hash\|param_hash) per (pipeline_id, scenario_id, item_id, site_id) — ReplacingMergeTree(computed_at), read with FINAL
PIPE_forecast_netting Netting step pipeline_id, forecast_week, item_id, site_id, item_name, stat_forecast_qty, return_forecast_qty, firm_demand_qty, total_netted_qty, return_netted_qty, stat_netted_qty, maint_netted_qty, consumption_enabled, netting_direction, is_past
PIPE_supply_orders Supply step pipeline_id, scenario_id, item_id, site_id, order_type, qty, release_week, arrival_week, status
PIPE_supply_inventory_projection Supply step pipeline_id, scenario_id, item_id, site_id, week, projected_inventory, demand, supply_received, shortage, safety_stock
PIPE_supply_exceptions Supply step pipeline_id, scenario_id, item_id, site_id, week, exception_type, exception_type_label, severity, qty
PIPE_supply_capacity Supply step pipeline_id, scenario_id, resource_id, resource_type, resource_name, week, capacity_qty, used_qty
PIPE_meio_results MEIO step pipeline_id, scenario_id, item_id, site_id, committed_buffer, fill_rate, wait_time
PIPE_series_characteristics Characterization step pipeline_id, scenario_id, item_id, site_id, item_name, site_name, has_seasonality, has_trend, is_intermittent, adi, zero_ratio
PIPE_series_hashes Demand hashing Demand-only / tenant-global data_hash per (item_id, site_id) — no pipeline_id column
PIPE_backtest_fingerprints Backtesting step Backtest fingerprint per (pipeline_id, scenario_id, item_id, site_id) — ReplacingMergeTree(computed_at), read with FINAL
PIPE_process_log All steps pipeline_id, step_name, status, started_at, ended_at, rows_processed, error_message

The CH schema is defined in files/DDL/ch_schema.sql. Use ReplacingMergeTree for deduplication on re-run (supply steps clear-and-rewrite their partitions using ALTER TABLE ... DROP PARTITION before inserting).

The Dask Parallelisation Layer

Forecasting and distribution fitting are the two most CPU-intensive pipeline steps. They use Dask for parallelisation:

# files/utils/orchestrator.py (simplified)
client = Client(n_workers=os.cpu_count())   # local Dask cluster

futures = []
for chunk in chunks(all_series, chunk_size):
    future = client.submit(run_forecast_chunk, chunk, method_list)
    futures.append(future)

results = client.gather(futures)   # blocks until all chunks done

Each chunk is a batch of series processed by a single Dask worker. Results are gathered and bulk-inserted into PostgreSQL.

Neural models and GPU

Neural models (NHITS, NBEATS, PatchTST, TFT, DeepAR) run in a separate pass using NeuralForecast's PyTorch-based parallelisation, bypassing Dask. They use GPU if available (CUDA detected automatically). Without GPU, they fall back to CPU — which is slow for large portfolios.

Dask teardown noise

On Windows, Dask workers emit harmless CommClosedError / StreamClosedError / ConnectionResetError messages as the local cluster shuts down after each pipeline step. These are suppressed in files/utils/orchestrator.py by a _DaskTeardownFilter applied to the distributed logger. If you see a single INFO Dask teardown complete line in the logs, this is normal — the suppressor is working.

The Rust MEIO Optimizer

The MEIO optimizer (files/meio_optimizer_v2.pyd) is a compiled Rust extension loaded via PyO3:

# files/meio_runner.py
from meio_optimizer_v2 import MeioOptimizer   # compiled .pyd / .so

optimizer = MeioOptimizer(items, echelons, distributions, targets)
result = optimizer.optimize(max_iterations=50_000)

Why Rust? The greedy marginal-value algorithm over 600 SKUs x 3 echelons x 50,000 iterations is approximately 10^9 floating-point operations. In Python this takes over 10 minutes; in Rust with Rayon (data-parallel library) it takes under 5 seconds.

The Rust code:

  • Uses rayon::par_iter() for parallel marginal-value computation across items
  • Uses lock-free atomic operations for the shared budget counter
  • Exposes a single optimize() function to Python via PyO3 bindings

Frontend Bundle Architecture

The React SPA uses Vite code splitting to keep initial load fast:

graph TD
    App["App.jsx — router + providers"]
    App -->|lazy chunk| D["Dashboard.jsx"]
    App -->|lazy chunk| TSV["TimeSeriesViewer.jsx"]
    App -->|lazy chunk| MEIO["MeioScenarios.jsx"]
    App -->|lazy chunk| EW["ExceptionWorkflow.jsx"]
    App -->|lazy chunk| SP["SupplyPlan.jsx"]
    App -->|lazy chunk| CF["CausalForecasting.jsx"]
    App -->|eager| Auth["AuthContext"]
    App -->|eager| Pipeline["PipelineContext"]
    App -->|eager| Exception["ExceptionContext"]

Each lazy(() => import(...)) creates a separate Vite chunk downloaded only on first navigation to that route. ExceptionContext is eager — always loaded — because the navigation overlay must be available on all pages.

Process Logging and SSE Streaming

Pipeline steps log progress in real time to the frontend via Server-Sent Events:

sequenceDiagram
    participant FE as Frontend
    participant API
    participant Proc as Background Process
    participant DB as process_log table

    FE->>API: POST /api/process/run?step=forecasting
    API->>Proc: spawn subprocess (asyncio)
    API-->>FE: {process_id: "abc123"}
    FE->>API: GET /api/process/abc123/stream (SSE)
    loop Every log line
        Proc->>API: stdout line to ListHandler buffer
        API-->>FE: data: line text
    end
    Proc->>DB: INSERT INTO process_log (step, status, started_at, ended_at)
    API-->>FE: data: done event

The ListHandler is a Python logging.Handler that appends log records to an in-memory list. The SSE endpoint yields from this list, then waits for new records. When the process exits it sends a terminal event and closes the stream.

Security Boundary Summary

Boundary Protection
Browser to API JWT Bearer token in Authorization header; 401 if missing/expired
Pipeline scripts Run as same OS user as API; no additional auth (trusted internal)
Tenant DB isolation account_id in JWT maps to connection string in master DB; impossible to access another tenant's DB without their JWT
SuperAdmin operations Separate role check in admin.py; only role="superadmin" can provision tenants
Token revocation revoked_tokens table checked on every request