Repository navigation
[feat] provider: first-class local vLLM-Omni backend (image/video/TTS/ASR) on consumer devices #224
Description
Motivation
OpenDirector is local-first by design — projects stay on your machine, and generation uses the provider APIs or self-hosted model services you choose. The launch demo already used vLLM-Omni to serve video generation (FastH3), and #205 extends the audio side (voices for vLLM-Omni and MOSS).
On the vLLM-Omni side, an RFC was just filed to make consumer-device self-hosting first-class — a full multi-stage backend (diffusion image/video + TTS + ASR) behind one localhost endpoint on an RTX laptop or an Apple Silicon Mac:
- vLLM-Omni RFC: [RFC]: Consumer-device local serving (RTX laptops, Apple Silicon) — backend for OpenDirector vllm-project/vllm-omni#10007 (shared endpoint items:
/v1/audio/transcriptions, CORS + capabilities, min-VRAM tiers) - Sub-issue RTX laptops (8–16 GB, Linux/WSL2): Local serving on RTX laptops (8–16 GB, Linux/WSL2) — sub-issue of #10007 vllm-project/vllm-omni#10008
- Sub-issue Apple Silicon (MPS, experimental): Local serving on Apple Silicon (MPS, experimental) — sub-issue of #10007 vllm-project/vllm-omni#10009
This issue tracks the OpenDirector side of that work so the pairing works end-to-end out of the box.
Proposed work items
| ID | Item | Scope | Notes |
|---|---|---|---|
| V1 | vLLM-Omni service preset in Settings → Model connections: endpoint mapping for chat (/v1/chat/completions), image (/v1/images/generations, /v1/images/edits), video (/v1/videos), TTS (/v1/audio/speech), ASR (/v1/audio/transcriptions when available), with 127.0.0.1 defaults. |
M | Extends the existing service/connection model; ASR endpoint lands via vllm-omni#10007 S1 |
| V2 | Capability detection and graceful degradation: probe which stages the configured vLLM-Omni server actually serves (capabilities endpoint, vllm-omni#10007 S2) and surface per-stage availability in the UI instead of failing generation attempts on unconfigured stages. | S–M | Also benefits other partial-configuration setups |
| V3 | Docs: "Run fully local with vLLM-Omni" guide — server startup for an RTX laptop (Linux/WSL2, pointing at #10008 profiles once published) and Apple Silicon status (experimental, #10009); localhost/CORS notes. | S | New page under docs/guides/; cross-link from the user guide's Model connections section |
| V4 | Validate harness: end-to-end acceptance — idea → script → image/video → TTS → export on (a) a single 8/12/16 GB RTX laptop and (b) an M-series Mac (diffusion/TTS local, text stages as available). | M | Follows the [validate] convention; needs community hardware — see help wanted |
Help wanted
- Owners of RTX 30/40/50 laptops (8/12/16 GB) and Apple Silicon Macs: report hardware, OS, and results on V4 once the vLLM-Omni profiles land; early smoke results welcome before that.
- Comments on V1 endpoint coverage — anything else the model-connection preset should map?
Cross-references: vllm-project/vllm-omni#10007, #10008, #10009; related here: #205 (audio voices), #107 (Q4 roadmap).
Activity
Xingkai98 commentedon Nov 10, 2026
I'd like to take V1 + V2 as one series (V1 first; happy to split). V1: a vLLM-Omni local-service preset mapping chat/images/videos/speech to 127.0.0.1 — ASR waits on vllm-omni#10007 S1. V2: stage-level capability probing, falling back to per-stage detection when capabilities aren't advertised, so unconfigured stages degrade instead of throwing. Fixtures + tests first. Could you assign these to me?
FayeSpica commentedon Nov 10, 2026
Thanks, @Xingkai98 assigned.
please collaborate with vllm-project/vllm-omni#10007.
Feel free to ask any questions.
bread-lxy commentedon Nov 10, 2026
Hi, I'd like to contribute to V2 as well. I already have a local implementation of capability discovery, the model settings UI and checks before generation. It keeps existing connections usable when a service doesn't expose capabilities, and preserves accepted tasks when capabilities or settings change. I'm also working on the corresponding S2 endpoint in vllm-project/vllm-omni#10007.
The two applications have been tested together over local HTTP without loading model weights. I'm finishing the input checks and Chinese/English UI wording, then I'll open a PR. Real-model validation is still pending.
@Xingkai98 I saw you've been assigned V1 and V2. Can we compare our approaches and agree on how to split the V2 work? I'd like to be included in V2 too, and make sure we use the same capability API on both sides.
FayeSpica commentedon Nov 10, 2026
Thanks, @bread-lxy! You’re both assigned now. Feel free to coordinate here on how you’d like to split the V2 work.
Xingkai98 commentedon Nov 10, 2026
@bread-lxy Thanks for the quick reply, and for laying out where your code already stands — that makes this much easier to split.
If it works for you, I'd suggest:
/v1/omni/capabilitiesendpoint. Your implementation is further along, so it would be a waste for us to write two versions of the same thing.127.0.0.1, with ASR mapped but flagged unavailable until vllm-omni#10007 S1 lands), plus the bilingual copy and tests.To keep us out of each other's files: you'd keep
integrations/model_capabilities.py,domain/capabilities.pyandModelSettings.tsx; I'd takeproviderPresets.ts,ModelServiceForm.tsxand the i18n files.application/model_catalog.pyis the one we both need — could we split it as you owningcapabilities()and me owning the preset resolution, each reviewing the other's PR?If you'd rather write the contract yourself, I'm equally happy to review it instead — no strong preference either way. Glad to be working on this with you, and thanks @FayeSpica for the assignment.
bread-lxy commentedon Nov 10, 2026
Thanks @Xingkai98, that split works for me: I'll handle V2's capability discovery, display and submission checks, plus the Omni endpoint. You'll handle V1's local-service presets and endpoint mappings, including the ASR availability handling and related copy/tests.
Both draft PRs are open:
On the connection form you proposed taking: my V2 PR already adds automatic-check settings and per-model capability checks to
ModelServiceForm.tsx, with Chinese/English states and errors. These are capability-checking controls; the V1 service preset still needs to be added. Could you review those changes first? If they look reasonable to you, we can agree on how your preset work fits into the same form and translation files.Your split of
model_catalog.pyalso works for me: capability checks on my side, preset resolution on yours. For the shared API format and test cases, there's already an implementation in these PRs. I'm happy for you to review and improve that as part of the shared work you offered.Thanks @FayeSpica for assigning us both!