Skip to content

Score on the Voice Arena dictation benchmark; drop the llm_instruction length cap - #322

Merged
alexkroman merged 11 commits into
mainfrom
evals/voicearena-no-length-cap
Nov 19, 2026
Merged

alexkroman merged 11 commits into
mainfrom
evals/voicearena-no-length-cap

Conversation

@alexkroman

@alexkroman alexkroman commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Brings the public split of the Voice Arena dictation benchmark into the dictation-prompt eval, and removes the 2048-character llm_instruction cap from both the prompt search and the app, now that the API no longer enforces it.

Voice Arena in the eval

  • New voicearena source (VoiceArena/Dictation_Dataset, 151 clips with recordings, CC BY-NC 4.0, loaded at runtime and never committed).
    • Input side is the service's own STT transcript of each recording, fetched once and cached (voicearena.py). The STT already writes "$62,000" and "plan.startDate", so spoken_text would make the search spend itself on number and code formatting the rewrite never sees.
    • Rows are held out of train, going to dev and test only, so the benchmark judges the search rather than teaching it. Long-form clips are kept.
    • The default source is now nyra,voicearena, and sources combine with commas.
  • --verify-live plays recorded clips instead of say where a corpus has them. --rewrite-mode smart sends rewrite_mode the way the app's Smart setting does.
  • voicearena_baseline.py scores Blurt on the public set with the benchmark's own judge (amend-va 0.2.1). It attributes each failing clip to STT or to the rewrite, and has --instruction-file, --rewrite-mode and --host (EU zone).

No length cap

  • The cap is gone in both modes. Re-probed 2026-11-18: instructions of 2049 to 49152 characters are accepted and rewrite correctly in Fast and in Smart. Only the whole config part is limited (64 KiB).
  • Removed: the soft target, the word and sentence budgets, section trimming and the length gate.
  • Kept: the proposer still gates on safeguards, field names and scaffolding.
  • The app drops its cap too (commit 1fe9169). CleanupInstruction has no characterCap, and sendable(appending:) always returns the instruction, so a longer prompt ships as-is. The transcriber's over-cap fallback is gone.
  • Style profiles get a fixed 4000-byte editor limit (StyleProfileStore.characterLimit), instead of the ~390 bytes of headroom the old cap left. It's still far inside the service's 64 KiB config limit.

Smart gets its own cleanup instruction

  • CleanupInstruction.smartText is text plus two edits, derived with replacing: it keeps greetings and meaningful framing ("Hey,", "just letting you know"), which the aggressive opener rule was deleting, and deletes "kind of" only as filler. Fast keeps text unchanged.
  • Measured on the Voice Arena public set with the benchmark's own judge, two independent samples each. On Smart it's better on both measures: ZER 90.1 / 88.1 vs 87.4 / 87.4, and AMEND 0.16 / 0.20 vs 0.33 / 0.26. On Fast the same edits scored worse (75.5 / 76.2 vs 78.1 / 78.1), so the instruction is chosen per mode.
  • The transcriber reads the mode once per request, for both the instruction and the echo check. With enhanced transcripts off, Fast's instruction is sent.

Reliability fixes

  • Hugging Face row pages are cached on disk. Every run re-read about 40 nyra pages and hit the rows API rate limit, token or not. Voice Arena's rows aren't cached, because their signed audio URLs expire.
  • DSPy is pinned below 3.5. 3.5.0 breaks every openai/ model call through the LLM Gateway with "Not found"; 3.4.1 works.
  • Voice Arena recordings are kept on disk (about 100 MB, downloaded once at load). Their signed URLs expire within hours, and a 4-hour search crashed when --verify-live got a 403.
  • --verify-live can no longer crash a finished run. Any failure is reported and the run carries on, so --out is always written.

Test plan

  • scripts/check.sh (full: Swift build, all test suites, SwiftLint, periphery, iOS checks, invariants)
  • scripts/check.sh --portable: ruff format and check, pytest (152 tests, including new ones for the held-out split, multi-source loading, the STT cache, recorded-audio playback, rewrite_mode, page caching and the gate-only proposer), prettier and markdownlint
  • --dry-run on the real nyra,voicearena corpus
  • Small end-to-end run with --optimizer none through the LLM Gateway
  • voicearena_baseline.py full runs on the public set (US Smart, US Fast, EU Smart)

🤖 Generated with Claude Code

…truction length cap

Voice Arena (voicearena.com/dictation-bench) ranks Blurt against thirteen other
Mac dictation tools. Its public split, 151 clips with recordings, is now part of
the dictation-prompt eval, and the 2048-character instruction cap is gone because
the API no longer enforces it.

- corpus/voicearena: `voicearena` source (VoiceArena/Dictation_Dataset, CC BY-NC).
  Its input side is the service's own STT transcript of each recording (fetched
  once, cached), since the STT already writes "$62,000" and "plan.startDate". Rows
  are held out of train (dev/test only) and keep their long-form clips. Default
  source is now nyra,voicearena; sources combine with commas.
- live: --verify-live plays recorded clips instead of `say` where a corpus has
  them; --rewrite-mode smart sends rewrite_mode like the app's Smart setting.
- No length cap: re-probed 2026-11-18, llm_instruction of 2049-49152 chars is
  accepted in Fast and Smart; only the 64 KiB config part is limited. Removed the
  soft target, budgets, trimming and length gate; the proposer still gates
  safeguards, field names and scaffolding. The Swift CleanupInstruction cap is
  untouched; the CLI warns when a winner exceeds it.
- voicearena_baseline.py: scores Blurt on the public set with the leaderboard's
  own judge (amend-va 0.2.1, Claude Opus 5.5) and attributes each failure to STT
  or the rewrite. --instruction-file, --rewrite-mode and --host (EU zone).
- Cache Hugging Face row pages on disk: every run re-read ~40 nyra pages and hit
  the rows API's rate limit, token or not.
- Pin DSPy below 3.5: 3.5.0 breaks every openai/ model call through the LLM
  Gateway ("Not found"); 3.4.1 works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alexkroman-assembly and others added 3 commits November 18, 2026 16:26
`live` is already imported at module level; flagged by code scanning on #322.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dictation API no longer caps config.llm_instruction: the reference now
documents it as `string | null` with no maxLength, and a probe on 2026-11-18
accepted 2049-49152 characters in Fast and Smart. Only the whole `config` part
is limited (64 KiB). So a longer cleanup instruction can ship as-is.

- CleanupInstruction: remove characterCap and customStyleBudget. sendable(appending:)
  always returns the instruction, with no over-cap fallback to the service default.
- AssemblyAITranscriber: drop the unreachable over-cap guard and its log line.
- StyleProfileStore.characterLimit: a fixed 4000-byte editor bound per profile,
  instead of the ~390 bytes of headroom the old cap left. The Mac and iPhone
  editors and the engine's trim all read it. Only the active profile is still
  sent, now justified on its own terms rather than by the cap.
- Tests pin the new behaviour: the instruction is always sent, a style is trimmed
  only to the editor limit (by bytes, on whole characters), and the longest
  request stays far inside the config limit.
- AGENTS.md, the engine README and the eval's notes describe the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a run

A 4-hour Smart search crashed at its last step. The Voice Arena audio URLs are
signed and had expired by the time --verify-live played them (403), and the
error escaped, so --out was never written.

- voicearena.ensure_audio: download the 151 recordings once at load, while the
  URLs are fresh (~100 MB, kept beside the STT cache). Utterances point at the
  local files. The STT step saves what it downloads too.
- live: play local recordings without the network. Refresh any URL that is
  still a URL just before use, and tolerate the datasets-server failing there.
- run_live_verification: catch any failure, report it and carry on. The
  search's result outranks the live check.
- describe_length: no cap warning, since the app's cap is gone too.
- voicearena_baseline: play the local copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman alexkroman changed the title evals: score on the Voice Arena dictation benchmark; drop the llm_instruction length cap Score on the Voice Arena dictation benchmark; drop the llm_instruction length cap Nov 19, 2026
alexkroman-assembly and others added 7 commits November 18, 2026 17:30
Clip ids come from a third-party dataset and name the cached recordings, so an
id like ../../x could have written outside the cache. Keep [A-Za-z0-9_-] only
and check the path stays in the directory. Real ids (dict_0007) are unchanged,
so existing cached files still match. Flagged by automated security review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
502s and 500s failed three loads in one afternoon, each fine on a manual rerun
seconds later, and each threw a run away before its first model call. Retry
500/502/503/504 with the same logged backoff as a rate limit; a 4xx still fails
at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two suites that landed on main after this branch began still treated the
cleanup instruction as possibly nil (a redundant #require, a dead ?? ""),
which CI's warnings-as-errors build rejects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Smart search tied the shipped prompt on Voice Arena. Every row GEPA ranked on
and learned from was Switchboard conversation, so the dictation failures that
separate prompts there never reached it.

- --voicearena-in-search [SHARE] (0.5 when bare) releases that share of the
  held-out Voice Arena rows into the search. carve_validation splits them evenly
  between GEPA's valset and its reflection set. The rest stay in dev/test to catch
  overfitting. Off by default; nothing changes without it.
- --seed-file PATH starts the search from a prompt in a file (e.g. the last run's
  winner). It is gated like any proposal and scored on dev next to BASELINE.
- The split printout lists the valset on its own line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rewrite runs a different model per mode, and one instruction no longer fits
both. CleanupInstruction now has two:

- `text`, Fast's, unchanged.
- `smartText`, Smart's: `text` with two edits, derived with `replacing` so the
  difference is explicit. It keeps greetings and meaningful framing ("Hey,",
  "just letting you know") that the aggressive opener rule was deleting, and
  deletes "kind of" only as filler.

Scored on the Voice Arena public set (151 clips) with the benchmark's own judge,
two independent samples each:
- Smart: ZER 90.1 / 88.1 vs 87.4 / 87.4, AMEND 0.16 / 0.20 vs 0.33 / 0.26.
  No clip the old instruction passed in both samples failed in both.
- Fast: ZER 75.5 / 76.2 vs 78.1 / 78.1, so Fast keeps `text`.

The transcriber reads the mode once per request and uses it for both the
instruction and the echo check. With enhanced transcripts off no rewrite_mode is
sent, so the service runs Fast and Fast's instruction goes with it.

Eval: candidates.SMART_WINNER is derived the same way, and a test applies the
Swift edit constants to PRIOR_WINNER to keep the two byte-identical.
--baseline smart-winner makes a Smart search's bar Smart's own instruction.
voicearena_baseline gains --sample for independent repeat runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner

StreamingPCMPlayerTests' drain bound restated drainSlack and allowed 0.5 s for
the poll loop and the scheduler. On CI, with no output device, the drain waits
out its whole deadline, and a loaded runner took 2.85 s against the 2.75 s
bound. Read drainSlack itself (now internal) and allow 2 s. The test guards a
drain that never ends, which no margin of seconds lets through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman
alexkroman enabled auto-merge (squash) November 19, 2026 04:38
@alexkroman
alexkroman merged commit 18c2537 into main Nov 19, 2026
11 of 13 checks passed
@alexkroman
alexkroman deleted the evals/voicearena-no-length-cap branch November 19, 2026 05:07
@alexkroman alexkroman mentioned this pull request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants