Elastic validation — real engineering test results (2026-07-06/07)
Architecture: session-affinity gateway (milestones/elastic_avatar.py) — sessions pinned to
replicas via modal.Dict, spill-over policy (probe replica k+1 only when 0..k busy), demand-driven
wake, fully-passive status. Replicas = independently labeled Modal web endpoints running the
hardened v1.0.0 server. Burst harness: milestones/elastic_burst_test.py (real renders only).
Proven by execution
| Claim | Evidence |
|---|---|
| Scale-out from cold under overload | round 1: r2 woke on demand, served 5 real sessions (14 s renders) |
| Correctness under 6× overload | round 2 (post-fixes): 40/40 PASS, zero gate-race errors |
| Warm-fleet distribution | final: 9/9 PASS split {r0:5, r2:4} (elastic_burst_final.json) |
| Scale-in to zero | observed hands-off at 303 s after idle (300 s window) |
| Session affinity | ~100 sessions, zero cross-replica errors |
| No spurious scale-out | light load (round 0): 30/30 on r0, cold replicas never woken |
| Self-healing | r0 died mid-burst, watchdog+min_containers recovered it |
| GPU capacity resilience | 4 types validated by real render (gpu_matrix_results.json); deployed: prod [A100-80GB,H100,L40S,A100-40GB], overflow [L40S,A100-40GB,A100-80GB,H100] (cheapest-first) |
Bugs found by the tests (all fixed & committed)
- Warmup-gate race (brief
accepts_new_session:truebefore Phase-B) → sustained-true 45 s gate - Gateway wake held one request 40 min → blocked scale-in → short repeated pokes
- Watchdog too aggressive under load → 20 s × 8
- Real A100-80GB capacity shortage mid-test → GPU fallback lists (the shortage actually hit prod)
- Observer effect: HTTP status probes reset replicas' idle clocks →
gw/statuspassive by default - Single aborted wake poke doesn't start a container (Modal cancels) → sustained pokes
Cold-start engineering (2026-07-07) — the 9–50 min problem, fixed to the physics limit
| Posture | Cold→traffic-ready | Idle cost | Notes |
|---|---|---|---|
Warm replica (min_containers=1) |
~0 s | $1.95/hr (L40S) | instant absorb |
| Fast-boot + self-warmup (DEPLOYED) | 406 s measured (boot 91 s + private capture render) | $0 | both post-gate renders PASS clean |
| Full Phase-B warmup (old) | 9–17 min typ, 50 min outlier | $0 | replaced |
| GPU memory snapshot | NOT VIABLE (alpha) | — | snapshot-create exceeded 2.6 h (single-threaded-compile tax); never reached a restore |
| Compile-cache Volume | NOT VIABLE (2nd confirmation) | — | warm-cache boot 676 s vs 405 s baseline: build emits almost no cacheable artifacts (only flashinfer); Volume latency hurts. Real fix = maintainer bakes compile caches into the image at build time |
Mechanism: WAN_S2V_WARMUP_ON_BOOT=0 (skip Phase-B) + a self-warmup render inside the gate —
one tiny internal session pays PCG capture before the port opens. Exposing the port during capture
breaks clients (backend event loop blocks → long-polls die with ServerDisconnected — measured), so
readiness-by-construction is mandatory, not optional.
Honest limits
- Cold bring-up is now ~7 min (was 9–50): still exceeds sub-minute spikes — for those, keep a warm overflow replica or pre-wake predictively at the gateway; ~7 min covers sustained bursts well.
- replica_1 had one flaky boot (container died mid-warmup; first L40S pull) — gateway tolerates it (spill-over skips unready replicas) but wake-retry-on-boot-failure is future work.
- Modal LB cannot do session affinity itself — the gateway (or boson-serve) must own it;
max_containers=1per replica label is mandatory.
Conclusion (2026-07-07)
Cold 405 s → traffic-ready · warm ~0 s (renders ~10 s). The 405 s decomposes as 91 s boot (API-up — faster than the dev image's oft-quoted 131 s, which measured the same milestone on an image that could never render) + 314 s self-warmup (JIT/compile/CUDA-graph capture + one private render). Both snapshot and compile-cache shortcuts are ruled out by measurement; the remaining lever is the maintainer baking compile caches into the image at build time. Sub-minute burst absorption is a cost knob (warm replica, ~$1.95/hr), not an engineering gap.