feat(ai): stream agent replies token-by-token to the room
Closes the cross-language half of token streaming (perf-plan A3). On the CPU-only box perceived latency is time-to-first-token, so showing the reply as it generates makes a slow model feel live. - Agent: OllamaProvider.stream() runs on a worker thread; bridge relays cumulative previews as throttled (~5/sec) `_ai:"stream"` control frames, then a `done` frame clears the preview as the final persisted chat message is posted. Providers without stream() fall back to blocking complete(). - Rust client: new Net::AiStream variant + parse_ai branch; App.ai_stream map holds the in-progress text per agent; draw_chat renders it as a dim, italic preview bubble below history. Cleared on done and on agent leave. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -17,11 +17,12 @@ KV-cache quant) are no-ops here.
|
||||
on CPU); `token_budget` default 3000→2000 to fit. `--num-ctx`, `--num-thread`,
|
||||
`--num-predict` flags added. `num_thread` defaults to Ollama's own (= physical
|
||||
cores, 4 here); benchmark 4/6/8.
|
||||
3. **Token streaming.** *(partial — provider half done)* `OllamaProvider.stream()`
|
||||
now yields deltas from Ollama's `stream=True` chat endpoint. Still TODO (commit 2):
|
||||
have the agent emit `_ai:"stream"` delta frames and the Rust client render an
|
||||
in-progress bubble. On CPU, perceived latency is TTFT — this will make a slow
|
||||
reply feel live.
|
||||
3. **Token streaming.** *(done)* `OllamaProvider.stream()` yields deltas from
|
||||
Ollama's `stream=True` chat endpoint; the agent relays them as throttled
|
||||
(~5/sec) cumulative `_ai:"stream"` frames off a worker thread, and the Rust
|
||||
client renders a dim in-progress preview bubble (cleared by a `done` frame
|
||||
when the final, persisted message lands). On CPU, perceived latency is TTFT —
|
||||
this makes a slow reply feel live.
|
||||
4. **Keep model warm + single-flight.** *(partial)* `keep_alive` already 30m
|
||||
(prevents mid-session reload). `OLLAMA_NUM_PARALLEL=1` is a **server-side env**
|
||||
read by `ollama serve`, not settable from the agent — set it where Ollama is
|
||||
|
||||
Reference in New Issue
Block a user