Milestone 6 — Serving + snapshot lifecycle ✅ (stand-in)
Goal: validate the memory-snapshot serving lifecycle the real renderer will use. Code: ~/modal_examples/milestones/m6_renderer_snapshot.py.
What was built
A Modal @app.cls with enable_memory_snapshot=True:
@modal.enter(snap=True)— heavy model init on CPU (captured in the snapshot), no GPU access.@modal.enter(snap=False)— the single.to("cuda"), run on each restore.@modal.method() render()— invoked viaspawn(), returns bytes (stand-in for MP4).
Weights come from the M5 Volume (/weights/model); this file deliberately doesn't download (separation of concerns).
How it was validated (deployed + ran)
[snap=True] model loaded to CPU in 2.98s
[snap=False] moved to GPU in 0.59s
RENDER: b'MP4-STANDIN:the capital of France is Paris...'
VALIDATION PASSED: snapshotted Cls renders via spawn()->get().
The snap=True/snap=False split is GPU-safe (no CUDA access during snapshot) — the key correctness property.
Code review (separate subagent) — CHANGES NEEDED → addressed
| Finding | Resolution |
|---|---|
| "weights never downloaded" (HIGH) | Not a bug — weights are provided by M5's Volume. Documented the precondition explicitly so the file isn't read as standalone. |
| Smoke test proves the method runs, not a snapshot restore | Documented scope: validates the lifecycle mechanism only; proving an actual restore needs a 2nd cold boot + log check. |
| Stand-in understates real-scale gaps | Documented: does NOT validate snapshot size, GPU-transfer time, or max_inputs concurrency at the real 75 GB scale. |
Honest scope
This validates that the snapshot lifecycle is wired correctly. It does not prove snapshot feasibility at 75 GB (size limits, restore time) — that needs the real image. The snap split itself is verified GPU-safe.
Real production image validated on Modal (2026-06-30)
The real production image was deployed and run on a real H100:
- ~75 GB image builds & runs on Modal (lazy ImageFS); flash-attn FA3 + SGLang + CUDA 12.8 load.
- The real GPU inference server (FastAPI/uvicorn:
/healthz, a synthesize endpoint,/v1/stop) starts — and its own docstring describes the CPU-gateway / GPU-container split that matches §4.4 of this plan. - Real number: ~95.3 s to container + image-stream + framework-init + weight-verify (lower bound; model-load adds more).
- Fails closed at model load: the heavy upstream weights are NOT baked in the image — they're provisioned from a private weight store. This validates M5: the production image requires weights provisioned from a private store.
Remaining blocker (precise): credentials for the private weight store (a modal.Secret) so the weights can be provisioned into a Volume mounted at the model path. Then: full model-load cold-start + a real render.
Status: 🔶 image + server boot validated on real H100; full model-load gated on weight-store creds.