Skip to content

Adaptive CPU Autoscaling

Overview

The adaptive CPU autoscaler monitors system CPU utilization during long-running pipeline steps and dynamically scales the Dask worker count up when cores sit idle. This addresses chronic CPU underutilization during backtest, forecast, and distribution-fitting steps where the static worker count (set at cluster startup) leaves cores idle when the batch queue is deep.

Two Modes

Mode Applies to Behaviour
Scale mode Dask steps (forecast, backtest, distribution fitting) Samples CPU % via psutil.cpu_percent(). When utilization is sustained below autoscale_low_cpu_threshold for autoscale_sustain_sec seconds AND queued Dask tasks remain, calls cluster.scale(N) to add workers. Caps at autoscale_max_workers (default: physical core count). Also enforces a total-threads cap: workers × threads_per_worker ≤ autoscale_max_threads.
Monitor mode MEIO, supply, ProcessPool fallback Samples CPU only, never resizes (rayon pool is fixed at startup). Emits a recommendation at step end that diagnoses whether the step is CPU-bound or I/O/DB-bound.

Why Direct Adaptation is Safe

The Dask steps (forecast, backtest, distribution) submit ALL batches up-front as futures. The scheduler holds pending tasks in a queue. Calling cluster.scale(N) mid-run simply adds worker processes that drain the existing queue — no re-splitting of work is required.

Config Keys

Stored in cluster_config.advanced in the config_parameters table. These keys live in DEFAULT_LOCAL_CONFIG["advanced"] in files/utils/dask_cluster.py and flow through _merge_defaults() so existing DB rows get them without a migration.

Key Default Description
autoscale_enabled True Master switch. False → monitor-only everywhere
autoscale_min_workers 1 Minimum worker (process) count
autoscale_max_workers 0 Maximum worker count. 0 = auto = physical core count
autoscale_min_threads 0 Minimum total threads across cluster. 0 = unset
autoscale_max_threads 0 Maximum total threads cap. 0 = auto = logical core count
autoscale_low_cpu_threshold 50 CPU usage % below which the scaler scales UP (sustained)
autoscale_high_cpu_threshold 92 CPU usage % above which the scaler scales DOWN (sustained; shrink off by default)
autoscale_sample_interval_sec 5 How often to sample CPU
autoscale_sustain_sec 15 Must stay below/above threshold this long before acting
autoscale_cooldown_sec 20 Minimum gap between scale actions
autoscale_step 2 Workers added per scale-up event
autoscale_allow_shrink False Enable scale-down toward min_workers above high threshold

UI: the "⚡ Autoscale" Subtab

Located in the Processes → Cluster tab as the 5th subtab (alongside Overview, Workers, Tasks, Config). The AutoscalePanel component provides:

  • Master enable/disable checkbox
  • Three min/max control groups:
  • Workers (CPU processes): min/max worker count
  • Total threads: min/max total threads cap
  • CPU usage %: low (scale-up trigger) / high (scale-down trigger)
  • Advanced tuning section (collapsible): sample interval, sustain window, cooldown, step size, allow-shrink checkbox
  • Live effective-limit chips showing: base workers×threads, scaler range, and effective max workers (factoring the thread cap)
  • Save / Save & restart buttons (wired to the existing PUT /api/cluster/config endpoint — no new API)

A warning appears if low_cpu >= high_cpu (inverted thresholds).

Scaling Logic (Scale Mode)

Every autoscale_sample_interval_sec:

  1. Sample psutil.cpu_percent(interval=None) (non-blocking)
  2. Push sample into a rolling deque (window = sustain_sec)
  3. Compute mean of the window
  4. Scale up if: mean CPU < low threshold for sustain window AND remaining queued tasks > current workers AND current workers < max workers AND cooldown elapsed
  5. new_n = min(max_workers, current + step)
  6. If max_threads > 0 and new_n × threads_per_worker > max_threads: clamp new_n down
  7. Call cluster.scale(new_n)
  8. Scale down (optional, if autoscale_allow_shrink): if mean CPU > high threshold for sustain window AND current > min_workers AND cooldown elapsed
  9. new_n = max(min_workers, current - 1)
  10. Call cluster.scale(new_n)

Recommendation Engine

At stop() time, the scaler always emits a recommendation:

Condition Recommendation
Scale mode, low CPU, scaling occurred "CPU averaged X% — autoscaler raised workers up to N. Consider setting local.n_workers=N permanently to avoid ramp-up latency."
Scale mode, low CPU, hit cap "Hit worker cap (N) but CPU still only X%. Bottleneck is per-task (GIL/ML torch threads) or stragglers — reduce batch_size and/or lower threads_per_worker."
Scale mode, low CPU, no scaling "CPU averaged X% but no scaling occurred (queue drained too fast or caps reached). Consider a smaller batch_size or raise autoscale_max_workers."
Scale mode, high CPU "CPU saturated at X%. To go faster: reduce batch_size or raise autoscale_max_workers."
Monitor mode, low CPU "CPU averaged X%. The engine appears I/O- or DB-bound; adding cores unlikely to help. Profile DB/CH round-trips."
Monitor mode, high CPU "CPU saturated at X%. The engine is CPU-bound — consider raising the rayon thread count."
Healthy "CPU utilization healthy (mean X%, peak Y%). No change recommended."

Per-Step Integration

Step File Mode Log prefix
Forecast step_forecast Scale [AUTOSCALE:forecast]
Backtest step_backtest Scale [AUTOSCALE:backtest]
Distribution fitting step_fit_distributions Scale (Dask) / Monitor (ProcessPool fallback) [AUTOSCALE:distribution]
MEIO meio_runner.py Monitor only —
Supply supply_runner.py Monitor only —

MEIO wraps the Rust run_optimization_batch call; supply wraps the _call_engine function (the Rust run_supply_plan call). The rayon pool is fixed at startup in both cases, so no live resize is possible.

End-of-Run Summary

After each instrumented step, the autoscaler prints:

  1. A summary line: [AUTOSCALE:step] mode=scale cpu mean=42% peak=67% p95=55% samples=12 scale_events=[4→6]
  2. A recommendation line.

At pipeline end, a compact per-step table is printed alongside the existing WORKER TIMING summary:

============================================================
ADAPTIVE CPU AUTOSCALING (per step)
============================================================
Step             Mode     CPU mean   peak    p95 Events
------------------------------------------------------------
forecast         scale          42%    67%    55%       1
backtest         scale          38%    71%    52%       2
distribution     monitor        55%    80%    62%       0
============================================================

API

No new API endpoints. Autoscale settings are persisted through the existing PUT /api/cluster/config endpoint (which writes the full cluster_config dict, merged with defaults by set_current_config). The GET /api/cluster/config endpoint returns the config.

Caveats & Known Limitations

Default-on

autoscale_enabled defaults to True. Existing deployments will see live cluster.scale() on the next run after this code ships. On shared boxes intentionally under-provisioned (e.g. n_workers=2 on a 16-core host shared with ClickHouse/torch), consider explicitly setting autoscale_enabled=false or lowering autoscale_max_workers before the first run.

  1. Default-on: autoscale_enabled defaults to True (see warning above).
  2. Worker spawn cost: LocalCluster(processes=True) means each cluster.scale() spawns a worker process that re-imports the forecast stack (pandas/statsforecast/numpy) — several seconds of startup. Scaling only amortizes for long steps with many batches.
  3. No memory check: The scaler does not check psutil.virtual_memory() before scaling. On memory-constrained boxes, cap autoscale_max_workers conservatively.
  4. External distributed scheduler: When mode=distributed, the scaler auto-downgrades to monitor mode (no local _cluster handle to resize).
  5. Scale-down off by default: autoscale_allow_shrink defaults to False. High CPU is logged as a recommendation only; no automatic scale-down occurs unless explicitly enabled.
  6. MEIO/supply fixed pool: The Rust rayon thread pool is fixed at process start. Monitor mode diagnoses I/O-vs-CPU bound but cannot resize.

Common Mistakes

Mistake Consequence Fix
Setting autoscale_max_workers too high on a shared box Oversubscription against CH/torch Set to physical cores minus reserved cores
Setting low_cpu >= high_cpu Inverted thresholds; scaler acts on impossible band Ensure low < high; UI shows a warning
Expecting MEIO/supply to scale live Rayon pool is fixed; only recommendation emitted Tune meio_parallel_workers at startup instead
Setting autoscale_max_threads without considering threads_per_worker Effective max workers may be lower than expected Check the "Effective max workers" chip in the UI
Forgetting that autoscale_enabled defaults to True Unexpected worker spawning on first run Explicitly set to false if not ready

Implementation

Key files

File Role
files/utils/cpu_autoscaler.py CpuAutoscaler class, AutoscaleReport dataclass, background thread, recommendation engine
files/utils/dask_cluster.py DEFAULT_LOCAL_CONFIG["advanced"] with autoscale_* keys; _merge_defaults() backfills them
files/utils/orchestrator.py _make_autoscaler() helper; wraps forecast/backtest/distribution Dask loops; _format_autoscale_summary()
files/meio_runner.py Wraps Rust run_optimization_batch with monitor-mode scaler
files/supply_runner.py Wraps _call_engine (Rust run_supply_plan) with monitor-mode scaler
files/frontend/src/components/ClusterTab.jsx AutoscalePanel component + "⚡ Autoscale" subtab

See Also