Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion aai_cli/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ heavily-reworked commands with long bodies; small commands keep the inline

### Feature subsystems

- **`streaming/`** + `client.stream_audio` — v3 realtime API. Event callbacks run on the SDK reader thread and guard against `BrokenPipeError` (`stdio.silence_stdout()`) so a closed pipe never dumps a thread traceback.
- **`streaming/`** + `client.stream_audio` — v3 realtime API. Event callbacks run on the SDK reader thread and guard against `BrokenPipeError` (`stdio.silence_stdout()`) so a closed pipe never dumps a thread traceback. Under `stream --llm` the `run_chain` request runs outside `StreamSession._callback_lock` (so a second `--system-audio` source keeps streaming), so refreshes can finish out of order: `_publish` orders publication under that lock by the published transcript turn count (`_published_len`), dropping an answer that covers no more turns, and the closing flush compares against that published coverage rather than the claimed `_summarized_len`, so a refresh still in flight at Ctrl-C can't leave the `--save-dir` note short.
- **`core/dictation.py`** + **`core/signals.py`** + `commands/dictate/` — `assembly dictate`: headless dictation over the **Dictation API** (<https://www.assemblyai.com/docs/dictation>; `Environment.dictation_base`, one multipart POST `/v1/transcribe/live` per utterance — the JSON `config` part **must precede** the raw-PCM `audio` part or the server answers 400; up to 120 s). The response carries the verbatim `text` and the send-ready cleanup rewrite `llm_response` (best-effort: on `llm_error` it is null and `DictationResult.final_text` falls back to `text`); dictate prints the rewrite by default, `--verbatim` the transcript, and `--instruction`/`--rewrite-mode` map to `llm_instruction`/`rewrite_mode`. `--keyterm` is `keyterms_prompt` (`--word-boost` is a hidden deprecated alias). It needs no terminal: recording starts immediately and `dictate_exec._record` polls `signals.stop_on_terminate` between ~100 ms mic chunks for a SIGTERM, which finishes the utterance (clean exit 0) — so a hotkey tool like Hammerspoon can launch it as a background task and `kill -TERM`/`task:terminate()` to transcribe. SIGINT (Ctrl-C) still cancels (exit 130). Both boundaries (the stop latch, mic, HTTP) are injectable, so the suite never needs a real signal or microphone (`tests/test_dictate_exec.py` scripts the SIGTERM latch). Contrast `signals.terminate_as_interrupt` (used by `stream`/`agent`/`speak`), which routes SIGTERM into the *cancel* path instead.
- **`agent/`** — full-duplex voice agent (mic in, TTS out via `voices.py`).
- **`agent_cascade/`** + `commands/agent_cascade/` — `assembly agent-cascade`: the same live terminal conversation as `assembly agent`, but **client-orchestrated** — `engine.run_cascade` wires Streaming STT → the LLM Gateway → streaming TTS itself instead of talking to the Voice Agent endpoint, mirroring what the `agent-cascade` `assembly init` template does server-side. Reuses the agent slice's `DuplexAudio`/`AgentRenderer` and `core.client.stream_audio`/`core.llm.complete`/`tts.session.synthesize`; the three network legs are injected through `engine.CascadeDeps` (the `tts/session.py` seam) so the cascade — greeting, clause-level streaming TTS, barge-in — is unit-tested against fakes with no sockets/mic/speaker. The LLM leg is a deepagents graph **assembled** by `brain.py` (tools, backend, middleware, `build_graph`, plus the shared `SpeechDelta`/`ToolNotice`/`ApprovalPause` event types and tool-affordance vocabulary) and **driven** turn-by-turn by `streamer.py` (`build_streamer`, `graph.stream(stream_mode="messages")`) — the natural build-vs-drive seam, split so each file stays under the 500-line gate (the dependency is one-directional: `streamer` imports `brain`, never the reverse). **Context-window management is the graph's job, not the engine's** — `create_deep_agent` wires deepagents' own `SummarizationMiddleware` into the stack (summarize the oldest turns, offload the evicted history to a file), so the engine feeds the *full* untrimmed running history each turn and lets the graph compact it; the old client-side `text.trim_history`/`config.max_history` sliding window is gone from this path (`max_history` now only drives the hand-rolled `--show-code`/`assembly init` cascade, which doesn't use deepagents). The engine buffers `SpeechDelta`s, flushes complete clauses with `text.pop_clauses` (soft-separator clauses gated by `engine._MIN_CLAUSE_CHARS`), and synthesizes each clause with **streaming TTS** (`tts.session.synthesize(on_audio=…)`) so audio starts on the first frame instead of after the whole reply. The reply runs on a throwaway producer thread feeding a `queue.Queue` the worker drains under a monotonic deadline (the wall-clock backstop that replaced `_complete_within`), and an abandoned-on-timeout graph leg's langchain `ThreadPoolExecutor` worker is detached (`_detach_executor_threads_since`) so it can't wedge interpreter exit. A `ToolNotice` surfaces the "Searching the web…" affordance and drops any unspoken preamble. **The `write_todos` plan** (deepagents auto-installs langchain's `TodoListMiddleware`) is reassembled from the streamed call args by `plan.py`'s `TodoCollector` and surfaced as a `plan.TodoUpdate` (not a `ToolNotice`); `engine._consume` forwards it to `renderer.todos_updated` without gating speech, so the voice TUI shows an in-place `messages.TodoList` checklist (and `AgentRenderer` emits a `plan` NDJSON event / a stderr checklist) — a visible plan to go with the spoken "first I'll…, then I'll…". Under `-v` (`debuglog.active()`) `streamer._stream_graph` logs each accumulated assistant line, tool call, and tool result as it streams. **Front-end:** an interactive mic session in human mode runs a **voice-only Textual TUI** (`agent_cascade/tui.py`, `LiveAgentApp`) by default — there's no text input (you can't type to it), just a transcript + an animated voice bar tracking listening/thinking/speaking. It uses its own `banner` wordmark, `messages` widgets, and `tui_status.voicebar_markup`/`VOICE_FRAMES` — all modules that now live in `agent_cascade/`; the blocking `run_cascade` runs on a worker thread and reaches the UI through a `_TuiRenderer` (the `engine.Renderer` protocol) that hops each call onto the UI thread, and a quit calls `DuplexAudio.close` to end the mic iterator and unblock that worker. `_exec._should_use_tui` gates it: file/sample input, `--json`/`-o text`, and a non-TTY all fall back to the plain `AgentRenderer` line output. **`--files`** (on by default; `--no-files` opts out) swaps the brain's in-memory backend for a real-cwd, sandbox-capable `SandboxedShellBackend` (`aai_cli/agent_cascade/sandbox.py`): file ops behave as before (traversal-blocked `virtual_mode`), and because it implements `SandboxBackendProtocol` deepagents binds a *functional* `execute` that runs commands OS-sandboxed in the real cwd — `sandbox-exec` (SBPL) on macOS, `bwrap` on Linux, refused (never an unconfined fallback) on any other platform or with the sandbox binary missing; the OS sandbox blocks the network, confines writes to cwd (+ the temp dir), and read-denies credential stores (`~/.ssh`/`~/.aws`/…, `.env*`, `.claude/`). The policy renderers are pure and the subprocess/capability boundaries injected, so the suite asserts *what we'd run* with no real sandbox. `write_file`/`edit_file`/`execute` are gated via `interrupt_on` + an `InMemorySaver`. The write gate is **path-scoped (the allow/interrupt permission model), not all-or-nothing**: `write_gate.write_interrupt_on(config.auto_write_paths)` hands `write_file`/`edit_file` one `InterruptOnConfig` whose `when` predicate fires only for writes *outside* every `--auto-write` subtree, so a write inside one runs ungated while others still pause (empty paths gate every write, the prior behavior). The matching is a transparent posix path-segment prefix check — `--auto-write DIR` is a directory subtree, and reads stay ungated, so no glob/bulk-tool machinery is needed (and no reach into deepagents internals: `create_deep_agent(permissions=…)` is unusable here — it raises `NotImplementedError` against an execute-capable backend like our `SandboxedShellBackend`). A non-string or `..`-containing target fails safe to gating. `execute` stays a plain `True` (unconditionally gated) — a command isn't a single file path to scope. `--auto-write DIR` is normalized to a cwd-rooted virtual root in `_exec._normalize_auto_write` (rejecting `..`/`~`/whole-tree aliases) and requires `--files`. `streamer._stream_gated` detects the post-stream interrupt (`graph.get_state(config).interrupts`), asks an injected `Approver`, and resumes with `Command(resume=…)`, bracketing the human wait in `ApprovalPause` events so `engine._consume` suspends its reply deadline (`risk.py` surfaces a shell-risk warning on the prompt). The voice TUI supplies the approver via `agent_cascade.modals.ApprovalScreen` (`y`/`a`/`n`), which can *also* be resolved hands-free by voice: while a write awaits approval, `_consume` arms `_awaiting_approval` and `engine.on_turn` routes the next final transcript to `app.submit_voice_approval` → `ApprovalScreen.try_voice`, which applies `spoken_approval.spoken_decision` (an unambiguous affirmative approves, anything else rejects — fail-safe; destructive `risk.py`-flagged commands ignore the spoken answer and require a keypress). **Project grounding (independent of `--files`):** `_exec.run_agent_cascade` reads the launch directory's `AGENTS.md`/`CLAUDE.md` via `agent_cascade/project_context.load_project_context()` into `CascadeConfig.project_context`, which `brain.build_graph` threads into `prompt.build_system_prompt(..., project_context=…)` (appended as project background after the persona/tool guidance). `AGENTS.md` wins precedence, identical content (a symlinked `CLAUDE.md`) is de-duplicated, and the total is capped at `project_context.MAX_CONTEXT_CHARS`. It's read at the command boundary (not in `build_graph`) so the brain stays hermetic, and the `--show-code` path builds its own config without it. Headless runs auto-deny (`_exec._deny_writes`). `--files` also turns on durable per-project memory via deepagents' `MemoryMiddleware` (`memory=["./.deepagents/AGENTS.md"]`), distinct from the in-session `InMemorySaver`. The gateway-bound, sandbox-backed general-purpose subagent (deepagents' `task` tool) for delegating a focused subtask is **auto-added by deepagents** — we don't declare it. We only override its prose for a voice turn (a spoken-length summary, not the SDK's "complete answer" default) via a harness profile keyed by the gateway model's provider (`subagents.register_gp_subagent_profile`, called from `build_graph` so the deepagents import stays lazy — and kept off `brain.py`, which sits at the 500-line gate). It inherits the gateway-bound model, the sandboxed toolset, *and* the top-level `interrupt_on` (deepagents' `graph.py` merges the top-level config into the auto-added subagent), so a delegated `write_file`/`edit_file`/`execute` surfaces at the *parent* `get_state().interrupts` with no per-subagent restatement (so `_pending_writes` gates it too — verified by a HITL spike, locked in `tests/test_agent_cascade_subagents.py`). Reads (incl. `grep`) stay ungated.
Expand Down
33 changes: 29 additions & 4 deletions aai_cli/streaming/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,9 @@ class StreamSession:
# -inf so the very first finalized turn always produces an immediate summary.
_summarized_len: int = 0
_last_summary_at: float = float("-inf")
# Turn count of the answer last shown/kept. Refreshes run unlocked and can finish
# out of order, so an answer covering no more turns than this one is stale.
_published_len: int = 0
# First CLIError raised by an --llm refresh. The chain runs on the SDK reader
# thread, where raising would dump a thread traceback instead of reaching
# run_command, so the error is recorded (and warned once) there and re-raised
Expand Down Expand Up @@ -177,7 +180,7 @@ def _maybe_summarize(self, *, final: bool = False) -> None:
return
with self._callback_lock:
turns = len(self.transcript)
if turns <= self._summarized_len:
if turns <= self._refreshed_through(final=final):
return # nothing new since the last refresh
now = self.clock()
throttled = self.llm_interval > 0 and now - self._last_summary_at < self.llm_interval
Expand All @@ -204,9 +207,31 @@ def _maybe_summarize(self, *, final: bool = False) -> None:
f"[aai.muted]--llm refresh failed: {exc.message}[/aai.muted]"
)
return
# Hold the latest answer so --save-dir can write it as the .md note at the end.
self._last_answer = answer
follow(answer, turns)
self._publish(follow, answer, turns)

def _refreshed_through(self, *, final: bool) -> int:
"""The turn count a refresh must exceed to be worth running.

Mid-stream, a claimed refresh already covers its turns even while it is still in
flight. The closing flush compares against the last *published* answer instead: a
reader thread still inside the chain when Ctrl-C ends the run may never land.
"""
return self._published_len if final else self._summarized_len

def _publish(self, follow: FollowRenderer, answer: str, turns: int) -> None:
"""Show and keep ``answer`` unless a refresh over as many turns already landed.

Two --system-audio sources (or a request outlasting --llm-interval) can finish
refreshes out of order; checking and publishing under the lock keeps the panel,
the NDJSON stream, and the --save-dir note from regressing to an older answer.
"""
with self._callback_lock:
if turns <= self._published_len:
return
self._published_len = turns
# Held so --save-dir can write the latest answer as the .md note at the end.
self._last_answer = answer
follow(answer, turns)

def _audio_target(self, source_label: str | None) -> Path | None:
"""The WAV path this source tees to, if any.
Expand Down
Loading