Elastic validation — real engineering test results (2026-07-06/07)
Architecture: session-affinity gateway (milestones/elastic_avatar.py) — sessions pinned to
replicas via modal.Dict, spill-over policy (probe replica k+1 only when 0..k busy), demand-driven
wake, fully-passive status. Replicas = independently labeled Modal web endpoints running the
hardened v1.0.0 server. Burst harness: milestones/elastic_burst_test.py (real renders only).
Proven by execution
| Claim | Evidence |
|---|---|
| Scale-out from cold under overload | round 1: r2 woke on demand, served 5 real sessions (14 s renders) |
| Correctness under 6× overload | round 2 (post-fixes): 40/40 PASS, zero gate-race errors |
| Warm-fleet distribution | final: 9/9 PASS split {r0:5, r2:4} (elastic_burst_final.json) |
| Scale-in to zero | observed hands-off at 303 s after idle (300 s window) |
| Session affinity | ~100 sessions, zero cross-replica errors |
| No spurious scale-out | light load (round 0): 30/30 on r0, cold replicas never woken |
| Self-healing | r0 died mid-burst, watchdog+min_containers recovered it |
| GPU capacity resilience | 4 types validated by real render (gpu_matrix_results.json); deployed: prod [A100-80GB,H100,L40S,A100-40GB], overflow [L40S,A100-40GB,A100-80GB,H100] (cheapest-first) |
Bugs found by the tests (all fixed & committed)
- Warmup-gate race (brief
accepts_new_session:truebefore Phase-B) → sustained-true 45 s gate - Gateway wake held one request 40 min → blocked scale-in → short repeated pokes
- Watchdog too aggressive under load → 20 s × 8
- Real A100-80GB capacity shortage mid-test → GPU fallback lists (the shortage actually hit prod)
- Observer effect: HTTP status probes reset replicas' idle clocks →
gw/statuspassive by default - Single aborted wake poke doesn't start a container (Modal cancels) → sustained pokes
Honest limits
- Cold bring-up (observed 9–50 min, host-cache dependent) exceeds short bursts — cold elasticity only helps sustained bursts; for spiky traffic, pre-warm (§18 readiness gate) or keep a warm pool.
- replica_1 had one flaky boot (container died mid-warmup; first L40S pull) — gateway tolerates it (spill-over skips unready replicas) but wake-retry-on-boot-failure is future work.
- Modal LB cannot do session affinity itself — the gateway (or boson-serve) must own it;
max_containers=1per replica label is mandatory.