Video generation — findings (2026-07-01 → 07-03)

Investigation of serving the wan-s2v / higgs-avatar speech→video model on Modal.

✅✅ FINAL RESOLUTION (2026-07-03): fMP4 avatar streaming WORKS on Modal via the official release

Base everything on boson-ai/higgs-avatar release v1.0.0 (commit 5c0cc03, image higgs-avatar-inference:a100-v1.0.0-5c0cc03). The engine-session path that is broken in the Jun-2 dev image (warlockee/wan-s2v:sglang-v3-selfcontained, dossier below) is fixed in the release.

Proven on Modal (A100-80GB), full boson-serve flow through the public URL: create → audio → audio/end → init.fmp4 (1240 B in **3.1 s**, no more 425-forever) → fragments → file.mp4 (559 KB, valid) — exit 0. Endpoint: https://bosonai--avatar.modal.run (stable label URL), deploy: milestones/wan_higgs_v1.py, client: milestones/v1_e2e_client.py.

Getting the release image onto Modal (totoro S3 is VPN-internal; Modal can't reach it): download tarball on a boson-network host (curl sigv4, resumable, ~27 MB/s) → skopeo copy docker-archive:… docker:// to a private Docker Hub mirror (orzeai/higgs-avatar-inference; docker load wedged twice — skopeo needs no docker daemon) → Image.from_registry(..., secret).

Two Modal-specific gotchas (both handled in wan_higgs_v1.py):

1. Strip /usr/local/cuda/compat from LD_LIBRARY_PATH — the image's forward-compat driver libs shadow Modal's injected GPU driver → torch sees no CUDA → backend crash-loops (torch.cpu has no attribute get_device_properties).
2. Gate traffic on FULL endpoint warmup, not replica_ready (since 2026-07-03 this is enforced in wan_higgs_v1.py itself — warmup-gated forwarder port — so clients no longer need to gate) — replica_ready:true appears before Phase-B per-bucket warmup finishes (~436 s × 3 buckets ≈ 17–22 min on A100). A session created during warmup queues behind it (looks like the old 425-forever). Wait until worker.busy=false ∧ queue_depth=0 ∧ no active sessions ∧ ≥3 done (exactly the release runbook's "add replica to LB only after warmup"). Also: this profile is single-session — don't run concurrent clients against one replica; abandoned concurrent sessions can wedge the lane (restart the replica to clear).

Fallback that also works today: the diffsynth-backed shim (wan_avatar_shim.py) — same API surface, proven PASS — useful if a replica must run on images without the fixed engine.

Cold-start numbers — don't conflate these:

✅ RESOLVED: offline video generation works (via the maintainer's own pipeline)

Use AvatarPipeline.synthesize_video_from_audio(audio_path, ref_image_path, out_dir, run_id, ...) (what gradio_avatar_all_in_one.py drives) — in-process, no HTTP client to hand-roll. Proven: a real 514 KB .mp4 produced from a ref image + 6.4 s audio (load ~52 s, gen ~44 s).

The "returns a path but writes no file" mystery = a silent VAE-decode OOM. The DiT runs fine (~40 GB), but the VAE decode grabbed 27.77 GiB in one shot (non-tiled, full spatial tensor) → CUDA OOM on the 80 GB H100 (model already ~50–64 GB) → zero frames written, yet the pipeline still returned the intended path. Notably the 27.77 GiB was identical at 864×1536 and 480×832, i.e. resolution-independent — a non-tiled decode.

Fix = WAN_LOW_RAM=1 (their documented flag; the config comment on height:1536 literally says "Needs WAN_LOW_RAM=1 for comfortable VRAM headroom") → tiled/chunked VAE decode → fits → MP4 written. Also set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Harness: milestones/wan_avatar_pipe.py. Implication for the design: single-GPU VAE decode has a memory ceiling; large frames need WAN_LOW_RAM=1 (tiling) or the streaming/per-block path.

✅ RESOLVED: fast provisioning + all 74 endpoints reachable via the URL (2026-07-02)

Measured through the live *.modal.run URL (evidence: PROVISION_VERIFICATION.md):

• Cold provision 131 s (URL wall-clock to first HTTP 200) / 144 s container-internal — < 3 min.
• Warm provision ~0.4 s — < 1 min.
• All 74 endpoints served (curl returns a real HTTP code for every one).

Three fixes made this work:

1. --host 0.0.0.0 — sglang's ServerArgs.host defaults to 127.0.0.1 (localhost); Modal's web proxy can't reach that → every URL request 303'd/timed out while localhost worked internally. run_server.sh forwards "$@", so run_server.sh --host 0.0.0.0 fixes it.
2. WAN_SGLANG_DISABLE_PREWARM=1 — drops the 3–5 min batched-prewarm tax → boot ~130–145 s.
3. No WAN_SGLANG_WARMUP=1 — its startup compile hits the VAE shape bug and never becomes ready (Modal then retry-loops containers). Compile is deferred to first request instead. Config also keeps WAN_SGLANG_SESSION_POOL_SIZE=1 (working sessions) + WAN_LOW_RAM=1. Note: 11 endpoints returned 000 on a sequential sweep — a single heavy handler (/get_weights_checksum ~10 s, session-creates) blocks the single GPU worker and cascades timeouts; re-tested in isolation (health-gate between calls) they all return 200. Harness: wan_web_stream.py (endpoint), wan_boot_measure.py, wan_api_test.py, wan_api_retest.py.

❌ BLOCKED ON MAINTAINER: engine-session generation is broken in this image build (2026-07-03)

Symptom (reproduced from boson-serve's e2e): create → audio → audio/end all 200, then init.fmp4 returns 425 forever; status ends {state:done, init_ready:false, fragment_count:0}; file.mp4 404. The server does long-poll (each init.fmp4 request held ~30 s server-side, honors wait_ms); the 303s seen externally are Modal's proxy on requests held >150 s — a symptom, not the cause.

Root cause location (from server logs): every engine session crashes in the maintainer's model code:

[WanS2VCausalDenoisingStage] Error during execution: 'NoneType' object has no attribute 'shape'
  model_s2v.py:343 run_blocks_with_audio → transformer_block_s2v.py:187  num_frames = temb.shape[1]

temb=None comes from the ref-pass in denoising_stage.py (~line 1785): prompt_embeds / _build_timestep_proj plumbing fails for engine sessions. The maintainer's own code carries a "loud diagnostic" comment for exactly this crash ("the block stack will see temb=None and crash") — a known failure mode of this in-development path (v3 code itself says it "replaces the broken Phase 2").

Exhaustive config matrix — ALL fail identically (each on H100, baked image):

What DOES work on the same image/server: the internal warmup sessions render 6+ blocks fine (bootstrap-warmup slot=0 done), and the in-process diffsynth path works end-to-end (real MP4 — see "offline video generation" above). So weights, GPU, DiT, VAE are all fine; only the engine-session request path (fMP4 stream + offline-chunks + v3 bench) is broken.

Also: docs/avatar-media-pipeline-v2.md references run_sglang_nonbs_backend.sh, which is not shipped in the image.

Ask to the image maintainer (warlockee/wan-s2v):

1. Fix engine-session temb/prompt_embeds plumbing (ref-pass in denoising_stage.py ~1785; crash at transformer_block_s2v.py:187) — their own diagnostic comment marks the spot.
2. Ship the documented run_sglang_nonbs_backend.sh (or update the doc to the real launcher).
3. Validate video_chat/test_sglang_nonbs_offline_e2e.py passes inside the published image — that's the acceptance test; our harness (milestones/wan_maintainer_e2e.py) runs it on Modal as-is.

Interim for boson-serve: use the offline path (works today, proven MP4) — either AvatarPipeline in-process or wan_s2v_server.py's /v1/generate; treat fMP4 streaming as blocked until a fixed image lands. Re-validate any new image with milestones/wan_maintainer_e2e.py

• milestones/wan_stream_matrix.py (one command each).

Streaming API (fMP4) — investigation notes (2026-07-01)

Correct server (architecture correction)

• Serve the raw sglang_native_wan_s2v.launch_server (via scripts/sglang_native_wan_s2v/run_server.sh, port 30000).
• Do NOT use inference/wan_s2v_server.py — that is a thin avatar wrapper exposing only /healthz + /v1/generate. Dropped.

Full production API surface (confirmed via /openapi.json — 74 routes)

📖 Call examples + real responses: see API_EXAMPLES.md — every endpoint with a real curl command and the verbatim response captured against the live deployment.

• /v1/videos, /v1/videos/{id}, /v1/videos/{id}/content — OpenAI/Sora-style create-a-video (prompt + image, NO audio; model defaults to sora-2). Not the audio path.
• /avatar/video/sessions (+ /audio, /audio/end) — audio-driven, offline.
• /avatar/video/stream/sessions/{id}/ init.fmp4 | fragments | file.mp4 | status | profile — the golden-standard streaming (fMP4) surface. POST /avatar/video/stream/sessions body = StreamCreateRequest (prompt, ref_image_b64|ref_image_path, width, height, fps, sample_rate, block_frames, audio_frames_per_latent, num_inference_steps, ...). Audio pushed to .../audio as {seq, sample_rate, pcm_f32_b64} (PCM float32, base64).
• /avatar/video/stream/replica/status → {"ok":true,"replica_ready":true,"accepts_new_session":true}.
• /wan_s2v/session/*, /wan_s2v/stream/sessions/*, /wan_s2v/offline/sessions/* (same handlers, /wan_s2v prefix).

Required launch config

• WAN_SGLANG_SESSION_POOL_SIZE≥1 — otherwise create_v3: SessionPool not initialized and the v3 session APIs fail. Default 0 = "v3 disabled".
• WANS2V_SKIP_MODEL_CHECK=1 (weights are baked/symlinked, skip the staleness check).
• WAN_SGLANG_WARMUP=1 — compile at startup so /health is reliable (default 0 defers compile → 303 flakiness).
• Boot pays a prewarm tax: measured /health-ready at ~100–680 s (varies with pool bootstrap + compile).

What works

Server boots, all routes mounted, replica_ready:true; session lifecycle (create → audio → audio/end → status) all return 200; DiT executes (dit≈365 ms/block, 100+ ticks). Infra/plumbing is solid.

What is blocked: producing an actual streamed .mp4 (fragment_count: 0 every config)

Each compile config trips a different error in the custom wan-code model internals:

These depend on the exact compile flags + streaming block geometry (block_frames, audio_frames_per_latent, vae_temporal_upsample, resolution). Not a Modal/infra problem.

Needed to finish

The maintainer's known-good streaming config — the exact env (compile flags) + StreamCreateRequest params used in production for /avatar/video/stream/sessions. The 60 vs 30 CausalTurboVAE cache mismatch (with wav2vec compile off, DiT working) is the precise diagnostic to hand them. Harness ready: milestones/wan_stream_e2e.py (plug in the reference params + env and rerun).

Harnesses (in milestones/)

• wan_stream_e2e.py — drives /avatar/video/stream/sessions end-to-end (create → audio → file.mp4).
• wan_video_server.py — serves the raw server + dumps the full /openapi.json route surface.
• wan_stream_test2.py — starts the server + drives bench_v3.py (internal perf client).
• wan_web_stream.py — Modal web endpoint for the raw streaming server (pool + warmup enabled).
• m_*.py — image introspection helpers (routes, bench args, request schemas).