Milestone 8 — Spillover + back-pressure ✅
Goal: a controller that decides per render: in-house / Modal / shed, with hysteresis (no flapping) and a hard cap on Modal spend. Code: ~/modal_examples/milestones/m8_spillover.py.
What was built
A SpilloverController with:
- Two watermarks (
high/low) → hysteresis: spill abovehigh, drain back belowlow, hold state in the band (no flapping). max_modal→ hard cap on concurrent Modal renders (cost / blast-radius). Over the cap ⇒shed.reconcile(actual)→ lost-render guard: resets the in-flight counter to Modal's real count so a crashed/dropped render can't leak a slot forever.- Thread-safe (a
Lockaround the counter) — the M9 control plane calls it from many threads.
How it was validated (python3 m8_spillover.py)
ok: hysteresis holds across oscillation; drains and re-enters
ok: max_modal cap enforced; overflow sheds
ok: modal_done + reconcile free capacity (lost-render guard)
ok: rejects low>=high and max_modal<1
ok: 200 concurrent routes -> exactly 10 granted, peak inflight 10 (no race)
VALIDATION PASSED
Code review (separate subagent) — CHANGES NEEDED → fixed
| Finding (severity) | Fix applied |
|---|---|
Thread-safety race on _modal_inflight (HIGH) — check-then-increment with no lock lets the cap be exceeded |
Added a threading.Lock around the read-modify-write; 200-thread stress test proves exactly max_modal granted, no drift |
| Lost-render leak (HIGH) — a dropped render leaks a slot forever | Added reconcile(actual_inflight) lease/reset |
| No config validation (MED) | __post_init__ raises on low>=high or max_modal<1 |
| Weak tests (HIGH) | Added oscillation, exact-boundary, config-guard, and concurrency stress tests |
Status: ✅ validated (thread-safe, cap-enforcing, lost-render-guarded).