Command-line settings for the two user-facing programs: the coli CLI and the openai_server.py server. The engines underneath (colibri, glm53, kimi_k3, inkling, qwen36, qwen38, deepseek_v4, deepseek_v41, olmoe) are driven by environment variables: see ENVIRONMENT.md, which says which engine reads which.
Updated for the contribution based on upstream/dev @ 21e7a35 (argparse definitions in c/coli and c/openai_server.py). See MAINTAINING-DOCS.md to regenerate.
coli <subcommand> [flags]
Flags may also be given after the subcommand. Most flags map onto an engine environment variable before glm is launched (see the mapping table at the bottom).
| Subcommand | Purpose |
|---|---|
build |
Build/prepare the engine. |
info |
Print model / build info. |
plan |
Show the computed RAM/VRAM placement plan (--json for machine-readable). |
mirror |
Plan, stage, or verify a usage-ranked partial model mirror (see multidisk.md). |
doctor |
Environment/health check (--json report, --deep strict preflight). |
tune |
Measure and save the fastest quality-preserving execution profile for this machine/model. |
run "<prompt>" |
One-shot generation for the given prompt (positional, may be multi-word). |
chat |
Interactive REPL chat. |
serve |
Start the OpenAI-compatible HTTP server. |
web |
serve, then open the dashboard in the browser. |
stop |
Shut down a running coli serve and its engine (default port: the one coli setup configured, else 8000). |
setup |
One step: detect the hardware, recommend a model, build or fetch the engine (Vulkan/CUDA when usable), download with resume, write the run configuration, start. A rerun starts directly, once make has brought the engine up to date with the sources (a rebuild only when they changed, as after git pull). See AI_SETUP.md. |
start |
Start what coli setup configured (--background, --no-browser, --json). |
status |
What is set up, install progress, server state, URLs, API model id, tok/s of the last answer (--json). |
logs |
Tail of the background server log (-n N, --install for the install log). |
mcp |
Model Context Protocol server on stdio for AI coding assistants. See MCP_SERVER.md. |
cluster coordinator / cluster worker |
Run the local-cluster control plane, or an expert worker that serves experts to it. |
bench [tasks] |
Run benchmark tasks (--limit, --data). |
convert |
Convert an FP8 repo to a colibrì int4 snapshot. |
| Flag | Default | Maps to | Meaning |
|---|---|---|---|
--model |
$COLI_MODEL or built-in path |
SNAP |
Model snapshot directory. |
--ram |
0 (auto ≈ 88% free) |
RAM_GB |
RAM budget in GB for the expert working set. |
--ctx |
0 (auto) |
CTX (the family's context variable) |
Context length. Auto, for a chat server (serve, web, chat): the largest of 16384, 32768, 65536, 131072 and 262144 tokens whose conversation state takes at most a quarter of the memory up to 65536 (coding agents need 64k) and a tenth beyond, else the family's default (DeepSeek V4.1 and MiMo count the prompt's workspace with the state: V4.1 prefills a prompt in one forward); GET /v1/models reports it as context_length (docs/api.md, "Conversations longer than the context"). |
--cap |
0 (auto) |
<cap> argv |
Expert-cache cap (starting point; see CAP_RAISE). 0 lets the engine pick: 8 historically, 1 on Metal + macOS when the model volume measures fast (F_NOCACHE probe ≥ COLI_SSD_FAST_GBS, cached in <model>/.coli_ssd — #379). An explicit value always wins. |
--ngen |
1024 for run; for a chat server (serve, web, chat) none: a response may run to the end of the context |
NGEN |
Max tokens to generate. A reasoning model's turn is its reasoning plus its answer, so a chat server leaves the ceiling to the context (#382); --ngen sets one. |
--temp |
none (0=greedy; engine default 1.0) |
TEMP |
Sampling temperature. |
--topp |
0 |
TOPP |
Top-p filter. |
--topk |
0 |
TOPK |
Top-k filter. |
--repin |
0 |
REPIN |
Re-pin experts every N tokens. |
--policy |
quality |
COLI_POLICY |
quality | balanced | experimental-fast. |
--gpu |
None |
COLI_GPU(S) |
auto, none, or a device list like 0,1. |
--vram |
0 (auto) |
CUDA plan | Total VRAM budget in GB. |
--auto-tier |
off | resource plan | Automatically apply the RAM/VRAM placement plan. |
--no-tune-profile |
off | profile loader | Ignore a saved measured profile. |
--think / --no-think |
the model's default | enable_thinking in the chat request |
Whether the model reasons before answering. Off, the prompt closes the reasoning block and the answer starts at the first word. |
--effort |
the model's default | reasoning_effort in the chat request |
minimal, low, medium, high or xhigh: how long the model reasons, on the families that take it. |
--xdna |
off | COLI_XDNA=1 |
Experimental: run qualified GLM operations on the AMD XDNA2 NPU (reduced-precision BF16 path; output may differ). |
--cluster-workers |
$CLUSTER_WORKERS |
CLUSTER_WORKERS |
Comma-separated expert workers, host:port,.... |
--cluster-coordinator |
$CLUSTER_COORDINATOR |
worker discovery | Control-plane URL used to discover expert workers. |
serve
| Flag | Default | Meaning |
|---|---|---|
--host |
127.0.0.1 |
Bind address. |
--port |
8000 |
Port. |
--model-id |
$COLI_MODEL_ID or glm-5.2-colibri |
Model id reported by the API. |
--api-key |
$COLI_API_KEY |
Require this bearer token. |
--cors-origin |
none (repeatable) | Allowed CORS origin(s). A plain value replaces the default list; +ORIGIN adds to it. |
--allowed-host |
$COLI_ALLOWED_HOSTS or none (repeatable) |
Additional Host header accepted by the DNS-rebinding guard. |
--max-queue |
$COLI_MAX_QUEUE or 8 |
Max queued requests. |
--queue-timeout |
$COLI_QUEUE_TIMEOUT or 300 |
Seconds a request may wait. |
--kv-slots |
$COLI_KV_SLOTS or 1 |
Independent KV conversation slots (→ KV_SLOTS). |
setup (no common flags: --model here is a catalog id, not a directory)
| Flag | Default | Meaning |
|---|---|---|
--yes, -y |
off | Non-interactive: the recommendation and every default. |
--model ID |
the recommendation | Catalog model to install (--list shows the ids). |
--model-dir DIR |
none | Use a model already on disk instead of downloading one. |
--dir DIR |
~/colibri-models |
Folder for downloaded models. |
--backend |
auto |
auto, cpu, vulkan or cuda: the engine build. auto takes CUDA, then Vulkan, then the CPU. |
--no-gpu |
off | Same as --backend cpu. |
--host, --port |
127.0.0.1, 8000 |
Where the configured server listens. |
--no-start |
off | Stop after writing the run configuration. |
--background |
off | Start the server detached (log in the setup folder). |
--no-browser |
off | Do not open the browser. |
--reconfigure |
off | Choose again even if a setup is complete. |
--list, --all |
off | List the catalog against this machine and exit; --all also shows what does not fit. |
--via-windows |
auto |
WSL only: download through Windows' curl.exe (auto: when it measures much faster). |
--no-verify |
off | Skip the checksums of downloaded files. |
--json |
off | Machine-readable result; implies --yes and a background start. |
The run configuration, state and logs live in COLI_SETUP_HOME (default
~/.local/share/colibri, ~/Library/Application Support/colibri,
%LOCALAPPDATA%\colibri). COLI_SETUP_CATALOG names a JSON list of extra
catalog entries (a mirror or a private repository); HF_ENDPOINT and
HF_TOKEN are honoured by the downloads.
convert
| Flag | Default | Meaning |
|---|---|---|
--repo |
zai-org/GLM-5.2-FP8 |
Source FP8 repo. |
--ebits |
4 |
Streamed-expert bit width. |
--io-bits |
8 |
Resident (attention/dense/embed) bit width. |
--xbits |
0 |
Extra/override bit width. |
--no-mtp |
off | Skip the MTP speculative-draft head. |
bench: [tasks...] (positional), --limit 40, --data <bench dir>.
plan / doctor: --json.
tune: --prompt <text>, --tokens 16, --repeats 2,
--timeout 900, --min-gain 0.03. The command uses fixed-token replay and
only tests quality-preserving execution scheduling. For disk-backed MoE it also
tests smaller, planner-bounded RAM/cache allocations; output, hit rate, TTFT,
and tail-latency gates prevent a decode-only win from degrading real chat.
doctor: --deep strictly checks every safetensors header and tensor
layout, filename-declared shard completeness, required core tensors, an
optional model index, and runtime-equivalent size/header admission for
COLI_MODEL_MIRROR. It does not hash tensor payloads or load the engine.
Run directly (or via coli serve). OpenAI-compatible /v1/chat/completions.
| Flag | Default | Meaning |
|---|---|---|
--model |
$COLI_MODEL (required if unset) |
Model snapshot directory. |
--engine |
./glm |
Path to the engine binary. |
--host |
127.0.0.1 |
Bind address. |
--port |
8000 |
Port. |
--model-id |
$COLI_MODEL_ID or glm-5.2-colibri |
Model id in API responses. |
--api-key |
$COLI_API_KEY |
Required bearer token. |
--cors-origin |
none (repeatable) | Allowed CORS origin(s). A plain value replaces the default list; +ORIGIN adds to it. |
--allowed-host |
$COLI_ALLOWED_HOSTS or none (repeatable) |
Additional Host header accepted by the DNS-rebinding guard. |
--cap |
0 (auto) |
Expert-cache cap; 0 = engine default (8, or 1 on Metal + macOS + fast model volume — #379). Without --cap, a coli tune profile or --auto-tier, DeepSeek V4.1, MiMo, Qwen3.6 and Qwen3.8 take the slots the RAM plan fits in the memory available at start (coli plan prints them). |
--max-tokens |
0 (the context) |
The response ceiling: a request without max_tokens gets it, a larger one is lowered to it. 0: the context the engine holds. GET /health reports it as max_tokens, and the web UI starts from it. |
--max-queue |
$COLI_MAX_QUEUE or 8 |
Max queued requests. |
--queue-timeout |
$COLI_QUEUE_TIMEOUT or 300 |
Request queue timeout (s). |
--kv-slots |
$COLI_KV_SLOTS or 1 |
KV conversation slots. |
Tool calling is supported by GLM, DeepSeek V4, Kimi K3, Qwen3.6, and Qwen3.8; Inkling rejects active tool declarations and choices explicitly, and OLMoE does too unless COLI_TOOL_FALLBACK=1. See the per-engine API matrix. The opt-in COLI_TOOL_SALVAGE=1 env var recovers malformed GLM int4 tool calls; V4 uses its native DSML parser. Engine-specific runtime variables are listed in ENVIRONMENT.md; the server passes the environment through to the selected engine.
A flag and its mapped environment variable are two routes to the same engine knob. Precedence and coverage:
- For knobs with a flag (
--temp,--ctx,--ram,--topk,--topp,--repin,--cap,--ngen,--policy), prefer the flag — it's the supported surface. - For knobs with no flag (
COLI_METAL,PIPE,DIRECT,COLI_NO_OMP_TUNE,MLOCK,CAP_RAISE,KVSAVE,SEED,NUCLEUS, …), export the environment variable. - The CLI copies your whole environment through to
glm, so any variable you export is honored unless a flag explicitly overrides it.