Operations¶

Day-to-day operational procedures for running and maintaining Mirabelle.

Starting and Stopping¶

Start (development)¶

# Windows — starts both API and frontend
start.bat

The batch file: 1. Kills any existing Python/Node processes on ports 8002 and 5173. 2. Starts the API: uvicorn api.main:app --port 8002 3. Starts the frontend dev server: npm run dev

Start (production / background)¶

# API in background
cd C:/allDev/ForecastAI2026.01/files
nohup python -m uvicorn api.main:app --host 127.0.0.1 --port 8002 --no-access-log \
  > /c/allDev/ForecastAI2026.01/api_8002.log 2>&1 &

echo "API PID: $!"

Stop¶

# Find and kill the API process
netstat -ano | findstr :8002   # Windows
# Then: taskkill /F /PID <pid>

# Or kill all Python processes (use with care)
taskkill /F /IM python3.13.exe

Restart Clean¶

# Kill existing processes
taskkill //F //IM python3.13.exe
taskkill //F //IM node.exe

# Start fresh
cd C:/allDev/ForecastAI2026.01
start.bat

Wait ~10 seconds for the API to load its in-memory data cache before hitting endpoints.

Process Logs¶

Via the UI¶

Navigate to Audit Log (/audit) for a paginated view of all actions. For pipeline step logs, go to Process Runner (/processes) which shows job status and live log output via Server-Sent Events.

Via the API¶

# List recent process log entries
curl http://localhost:8002/api/process-log?limit=20 \
  -H "Authorization: Bearer <token>"

# List distinct pipeline runs
curl http://localhost:8002/api/process-log/runs \
  -H "Authorization: Bearer <token>"

# Get steps for a specific run
curl http://localhost:8002/api/process-log/{run_id}/steps \
  -H "Authorization: Bearer <token>"

# Get log tail for a step
curl http://localhost:8002/api/process-log/step/{step_id}/tail \
  -H "Authorization: Bearer <token>"

Via the log file¶

API output is written to api_8002.log in the project root. Monitor it:

tail -f /c/allDev/ForecastAI2026.01/api_8002.log

Running Pipeline Steps¶

Via the UI¶

Navigate to Processes (/processes).
Select the pipeline from the dropdown.
Select a step (ETL, Characterization, Forecast, etc.).
Click Run — the step starts as a background job.
The job log streams in real time via SSE.

Via the API¶

# Run the ETL step
curl -X POST http://localhost:8002/api/pipeline/run/etl \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"scenario_id": 1, "pipeline_id": 42}'

# Response: {"job_id": "uuid", "step": "etl", "status": "queued"}

Available steps:

Step ID	What it does
`etl`	Extract from source DB, load demand_actuals
`outlier`	Detect and optionally correct outliers
`characterization`	Classify all series (seasonality, trend, etc.)
`forecast`	Run all enabled forecasting models
`backtest`	Rolling-window backtesting
`backtest-forecast`	Backtest + forecast in one shot
`best-method`	Select best method per series from backtest results
`distribution`	Fit probability distributions for MEIO
`meio`	Run MEIO optimization
`supply`	Run supply planning

Run the full pipeline¶

curl -X POST http://localhost:8002/api/pipeline/run-all \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"scenario_id": 1, "pipeline_id": 42}'

Monitor a job¶

# Check status
curl http://localhost:8002/api/pipeline/jobs/{job_id} \
  -H "Authorization: Bearer <token>"

# Stream log output (Server-Sent Events)
curl http://localhost:8002/api/pipeline/jobs/{job_id}/stream \
  -H "Authorization: Bearer <token>"

Reloading the Data Cache¶

After any pipeline run completes, reload the API's in-memory cache:

curl -X POST http://localhost:8002/api/reload \
  -H "Authorization: Bearer <token>"

This: 1. Syncs missing parameter keys to the DB (non-destructive). 2. Clears all cached DataFrames. 3. Re-reads everything from PostgreSQL.

Response: { "status": "reloaded", "cache_sizes": {...} }

Adjusting Log Level¶

To see detailed debug output without restarting:

# Set to DEBUG
curl -X PUT http://localhost:8002/api/log-level \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"level": "DEBUG"}'

# Check current level
curl http://localhost:8002/api/log-level \
  -H "Authorization: Bearer <token>"

Accepted levels: DEBUG, INFO, WARNING, ERROR.

Killing Stuck Jobs¶

If a pipeline job hangs:

# List all jobs
curl http://localhost:8002/api/pipeline/jobs \
  -H "Authorization: Bearer <token>"

# Kill a specific job
curl -X POST http://localhost:8002/api/pipeline/jobs/{job_id}/kill \
  -H "Authorization: Bearer <token>"

# Reset all stale jobs (marks them as failed)
curl -X POST http://localhost:8002/api/pipeline/jobs/reset \
  -H "Authorization: Bearer <token>"

If the job PID is no longer alive but the job record is stuck in running, use the reset endpoint.

Database Maintenance¶

Vacuum / analyse¶

PostgreSQL auto-vacuum handles routine maintenance, but after large bulk inserts (pipeline runs), a manual analyse helps the query planner:

ANALYZE scenario.demand_actuals;
ANALYZE scenario.forecast_results;
ANALYZE scenario.meio_results;

Check table sizes¶

SELECT relname, pg_size_pretty(pg_total_relation_size(oid))
FROM pg_class
WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = 'scenario')
ORDER BY pg_total_relation_size(oid) DESC
LIMIT 20;

Clean up old forecast results¶

Old pipeline runs accumulate forecast data. Clean up by scenario:

-- Delete forecast results for old scenario runs
DELETE FROM scenario.forecast_results WHERE scenario_id = 99;
DELETE FROM scenario.backtest_results WHERE scenario_id = 99;
DELETE FROM scenario.series_characteristics WHERE scenario_id = 99;

Backup and Recovery¶

Backup¶

# Full tenant database backup
pg_dump -U postgres forecastai > forecastai_backup_$(date +%Y%m%d).sql

# Master DB backup (tenant accounts and superAdmins)
pg_dump -U postgres forecastai_master > master_backup_$(date +%Y%m%d).sql

Restore¶

psql -U postgres forecastai < forecastai_backup_20260412.sql

After restoring, reload the API cache:

curl -X POST http://localhost:8002/api/reload -H "Authorization: Bearer <token>"

Checking API Health¶

# Check the API is responding
curl http://localhost:8002/

# Check DB connectivity and data cache
curl http://localhost:8002/api/config \
  -H "Authorization: Bearer <token>"

# Check series count
curl http://localhost:8002/api/series?limit=1 \
  -H "Authorization: Bearer <token>"

# Check MEIO results
curl http://localhost:8002/api/meio/scenarios \
  -H "Authorization: Bearer <token>"

All returning HTTP 200 indicates a healthy deployment.

Port Conflicts¶

The API runs on port 8002 by default. If this port is in use:

# Find the process using 8002 (Windows)
netstat -ano | findstr :8002
taskkill /F /PID <pid>

# Or change the port
python -m uvicorn api.main:app --port 8003

# Update Vite proxy in files/frontend/vite.config.js:
# proxy: { '/api': { target: 'http://localhost:8003' } }

Common Issues¶

Symptom	Cause	Fix
`503 Service Unavailable` on startup	API still loading data cache	Wait 10–15s; the api.js interceptor auto-retries
`401` on all requests after restart	Stale JWT in browser	Clear localStorage, log in again
Forecast step hangs	Dask worker crash	Kill the job via API; check `api_8002.log` for traceback
Series table shows 0 rows	Pipeline not yet run, or wrong pipeline selected	Run the ETL + characterization steps first
MEIO returns empty results	`meio_optimizer.pyd` not built	Check that the Rust extension is present in `files/`
`DB not available` error	PostgreSQL is down or config.yaml is wrong	Check PostgreSQL service; verify `files/config/config.yaml`