Repository navigation
bench: realtime caption eval set + Ollama-MLX vs llama.cpp-Metal comparison - #28
PCcoding666 wants to merge 1 commit into
Conversation
Adds a 60-case eval set that mimics what on-device ASR actually emits (short turns, unfinished drafts, no punctuation, fillers, numbers, names, zh->en, context follow-ups) and a runner that scores any Ollama or OpenAI-compatible server on streaming TTFT / end-to-end latency, generation speed, instruction compliance, term preservation and chrF. Results on an M2 Pro 16 GB with the same Qwen3.5-4B weights: llama.cpp (Metal, Q4_K_M) and Ollama (MLX runner, nvfp4) are within noise of each other (p50 end-to-end 542 vs 534 ms, TTFT 146 vs 160 ms, chrF 55.8 vs 57.6, terms kept 97% vs 92%). The Qwen2.5-1.5B model that PR #27 bundles is 3x faster but drops to 90% compliance, 89% terms kept and chrF 47.7: it echoes inputs untranslated, turns 38% into 3.8% and 五十万 into fifty thousand. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da5105e178
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for pat in VIOLATION_PATTERNS: | ||
| if pat.lower() in low: | ||
| problems.append(f"preamble:{pat}"); break |
There was a problem hiding this comment.
The substring check flags legitimate translations as instruction violations in the committed results. C03 starts with “okay okay”, so translating it as “好的，好的” is not an added preamble, yet both runs for every engine are penalized. Likewise, the 1.5B engine's G06 outputs contain “the translation” while translating the source's discussion of translation latency, not introducing an answer. These false positives distort the reported compliance comparison. Make detection source-aware or use case-specific expectations, then rescore the stored results.
Useful? React with 👍 / 👎.
| out = clean(output) | ||
| problems = [] | ||
| low = out.lower() | ||
| for pat in VIOLATION_PATTERNS: | ||
| if pat.lower() in low: |
There was a problem hiding this comment.
When an engine emits thinking blocks or delimiter markers, clean(output) removes them before the violation checks run, making the <think> and 🔤 patterns ineffective. For example, both <think>I should translate this.</think>我们开始吧。 and 🔤我们开始吧。🔤 score as fully compliant for A01. This hides precisely the output-format failures the eval set claims to measure. Detect these violations on the raw response and reserve cleaning for reference and term scoring.
Useful? React with 👍 / 👎.
| for t in case.get("terms", []): | ||
| if t.lower() not in low: | ||
| missing.append(t) | ||
| for group in case.get("terms_any", []): | ||
| if not any(t.lower() in low for t in group): |
There was a problem hiding this comment.
For numeric preservation cases, substring membership counts corrupted values as preserved. Using the supplied G03 case, “Last quarter our retention rate went from 138% to 145%.” receives terms_kept: 2 because it contains 38 and 45, despite changing both numbers. The same issue affects numeric alternatives in terms_any. Match complete numeric values with appropriate boundaries and explicit accepted representations so the reported numbers/names retention metric does not reward these errors.
Useful? React with 👍 / 👎.
Summary
Answers "does the embedded llama.cpp path cost us anything versus Ollama's MLX runner on the same model?" with numbers, and measures the Qwen2.5-1.5B model that PR #27 currently bundles.
benchmark/realtime_eval_set.json: 60 cases in 8 categories shaped like real ASR output (short turns, half-sentence drafts, no punctuation + fillers, numbers/names, tech terms, long sentences, zh→en, context follow-ups), each with a reference and the terms that must survive.benchmark/run_engine_compare.py: runs any Ollama or OpenAI-compatible server (llama-server, etc.) with streaming, thinking off,num_ctx4096, and scores TTFT / end-to-end / tok/s, instruction compliance (preamble, echo, wrong script, over-long), term preservation and chrF.--appendmerges engines run at different times,--report-onlyrebuilds the report.benchmark/engine_compare_report.md+engine_compare_results.json: this run (M2 Pro, 16 GB, 2 runs × 60 cases per engine).Headline numbers (p50)
Same weights, different runtime: no measurable difference. The 1.5B model is fast but makes errors a caption translator cannot afford (
38%→45%became3.8%→4.5%,五十万becamefifty thousand, several inputs echoed untranslated).Note: in Ollama 0.40 both
qwen3.5:4bandqwen3.5:4b-mlxare the same safetensors/nvfp4 weights on the MLX runner, so there is no Ollama-GGUF column; the second Ollama column is a repeat run. Memory: Ollama 4.1 GB resident atnum_ctx4096 (7.6 GB at its 32K default); llama-server 0.95 GB phys + 2.6 GB GPU.🤖 Generated with Claude Code