Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.
Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bench/ — a 4-category × easy/medium/hard task matrix (shell, code, git,
multi) and a runner that drives the live TUI over tmux and grades each task by
a `podman exec` verify snippet (exit 0 == PASS), never by the model's
self-reported summary (which the weak CPU model fabricates). Tasks run in the
agent's real cwd with bare filenames so the suite measures task completion, not
the model's absolute-path discipline. Completion is detected off the viewport-
independent `is thinking…` footer (the TUI is full-screen, so capture-pane
scrollback is not chat history).
First baseline (qwen2.5:3b): 2/12 PASS, high variance. Surfaces the next
harness-addressable improvements — `<native>` tag leakage and bare
tool-call-as-text — now measurable against this suite. Findings doc updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the overloaded "text + no tool call = done" terminator that made the
weak CPU model stall mid-task or give up after a failing command. Termination
is now a structural `DONE:` text sentinel; a text-only turn is resolved by an
output-aware verdict (DONE: marker / unresolved non-zero exit / no action /
filler language) and re-prompted with an exit-code-aware nudge, bounded by
MAX_NUDGES on top of max_turns. On exhaustion the summary is honest rather than
echoing the model's false "run successfully" — it reports when no tool ran or a
command exited non-zero. Live-validated on qwen2.5:3b: the multi-step stall is
fixed (proj3 completes end-to-end, ground-truth confirmed); the nudge fires on
a 126; residual give-up is model-bound. 23 offline unit assertions pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live 3B-vs-7B command-entry results, observed failure modes (early stall,
give-up-on-error, tool hallucination), Goose/opencode loop-termination
research, and the output-aware dynamic-nudge-loop design to implement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Display-mirror hybrid (docs/plan-harness-visibility.md §2): native tool
calls now show up in the shared sandbox terminal again via inert `# `-
prefixed comment lines (comment-prefix = anti-double-run/anti-escape),
mirroring only each command. Chat de-flooded to opener + final summary.
write_file mkdir -p parent dir so relative/absolute paths both work
(fixes the regression where script creation silently failed). ui.rs
fmt_line returns Vec<Line> splitting on \n so multi-line agent output
renders as an indented block instead of one garbled row.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the Goose agentic harness across the codebase per
docs/spec-native-harness.md §3. Goose made N sequential model calls inside the
sandbox (slow on CPU-only hardware) and forced an in-container→host Ollama
gateway that tripped the rootless-Podman slirp4netns loopback bug.
- bridge.py: delete _run_goose/_goose_argv/_goose_present + GOOSE_* consts and
the present-cache; __init__ now takes harness="simple"/max_turns=5; granted
!task runs _run_simple until the native loop lands (Phase 2).
- __main__.py: --harness {native,simple} (was {goose,simple}); drop
--goose-max-turns, add --max-turns; default harness simple.
- app.rs: /ai start accepts native|simple (plain aliases simple) instead of a
bare plain flag; refresh harness comments.
- sbx.rs: remove the in-container Ollama gateway (Docker host-gateway / Podman
slirp4netns host-loopback) and the dk_bootstrap OLLAMA_HOST env — kills the
slirp4netns loopback bug; drop Goose comments.
- bootstrap.sh: drop goose from the prereq probe.
- bootstrap-ai.sh: remove the entire Goose install block, --no-goose flag,
GOOSE_INSTALLER_URL, host config writer, and goose_bin helper.
- sandbox-bootstrap.sh: remove the in-sandbox Goose binary install + config.
- spec-goose-harness.md: banner — harness portion superseded; Podman stays.
cargo check + py_compile clean. No Goose refs remain (headroom/ untouched).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The /ai agent held a single, un-shielded websocket with no retry. Any
close — server restart, ping/idle reap, laptop sleep, a transient blip —
ended the serve loop, so run_async returned and the process exited
silently: the agent dropped from the roster with no /ai stop and no
goodbye.
- run_async now wraps the connection in a backoff-reconnect loop (1s→30s,
resets after a healthy ≥30s session). The server frees our session+name
on drop, so each attempt re-runs SRP to mint a fresh token. Only Ctrl-C
/ process kill (KeyboardInterrupt / CancelledError, how /ai stop ends
us) breaks the loop.
- _serve shields each frame via _handle_frame so one malformed/poisoned
frame — or a handler error — can't unwind the loop; ConnectionClosed
and cancellation propagate up to the reconnect loop.
- Forgiving keepalive (ping_interval=20, ping_timeout=60) so a heavy
CPU-only Ollama generation doesn't trip a false drop.
Also adds docs/spec-native-harness.md: replace the heavyweight Goose
harness with a lightweight host-side Ollama-native tool-calling loop
(model runs host-side, only commands exec in the sandbox — the
slirp4netns→host-Ollama bug disappears), and a file-by-file plan to strip
all Goose integration. Supersedes the harness portion of
spec-goose-harness.md (Podman backend stays).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the goose-harness and sandbox-distros-GUI design specs, a consolidated
command reference and demo-reels plan, the AI-harness planning note, and
two new client themes (blue-orange, pink-red-gray).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Make the message-input bar part of the F5 resize cycle so its height is
adjustable without a sandbox (multi-line/wrapped inputs are now readable),
and let chat/clergy borrow height from the compose box when no terminal is
present. Add Option C masked sudo prompt that feeds `sudo -S` over stdin to
install/start Docker — the password never reaches chat, the PTY, or outbound
frames, and Docker Desktop on Linux is detected so no sudo is requested.
Fix a freeze where clicking the compose box entered layout-edit mode and
silently swallowed every keystroke: clicking the input bar no longer enters
edit mode, and typing any printable char now drops out of edit mode and types.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Design for replacing the two-scalar layout (pty_pct + roster_width) with a
binary space-partition pane tree so every pane is resizable on both axes,
fixing the "only roster width adjusts" limitation. Rolls a small in-repo tree
(no new dep) and drives the PTY grid from the terminal pane's actual Rect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the 14 hack-house scripts (bootstrap, lets-hack, host-room, smoke,
demo/film harnesses, connect/join, ensure-docker/vbox) into hh/scripts/
and fix their path resolution for the deeper location. Update cross-refs:
sbx.rs script consts, app.rs hints, CI, direnv .envrc, README, and docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agents now join under their model tag (e.g. "qwen2.5:3b" — model name +
parameter size) or profile label, derived at /ai start and tracked on
App.agent_name, so the roster shows what's actually answering. Updates the
broker re-grant + /ai stop revoke paths, the Python --name default
(falls back to provider.model), and the demo/bootstrap/README/docs refs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the VirtualBox sandbox design spec (headless 4th backend + share-an-
appliance GUI mode with detect-first install), the crypto pay-to-join gate
design, and the save/load PoC writeup with its demo/film driver scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Closes the cross-language half of token streaming (perf-plan A3). On the
CPU-only box perceived latency is time-to-first-token, so showing the reply
as it generates makes a slow model feel live.
- Agent: OllamaProvider.stream() runs on a worker thread; bridge relays
cumulative previews as throttled (~5/sec) `_ai:"stream"` control frames,
then a `done` frame clears the preview as the final persisted chat message
is posted. Providers without stream() fall back to blocking complete().
- Rust client: new Net::AiStream variant + parse_ai branch; App.ai_stream
map holds the in-progress text per agent; draw_chat renders it as a dim,
italic preview bubble below history. Cleared on done and on agent leave.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tier A/B/C wins for the CPU-only Ollama box (no GPU → optimize TTFT and
tokens/sec, not VRAM):
- Separate qwen2.5-coder provider for the sandbox `!task` path; chat keeps
the general model. Auto-selected when chat is Ollama and a coder build is
present, override with --code-model.
- OllamaProvider num_ctx default 8192→4096 (8192 was a GPU-mindset default
that inflates prefill/TTFT on CPU); expose num_thread; add --num-ctx,
--num-thread, --num-predict. token_budget default 3000→2000 to fit.
- OllamaProvider.stream() generator over Ollama's stream=True chat endpoint
(provider half of token streaming; agent/Rust rendering is a follow-up).
- Few-shot request→shell exemplars in SANDBOX_SYSTEM to anchor the small
model's fenced-command output.
- Matryoshka embedding truncation: OllamaEmbedder truncate_dim=256 (--embed-dim)
for faster pure-Python cosine and less RAM; query+stored share the dim.
- docs/ai-perf-plan.md records all 8 items with status and the server-side
env (OLLAMA_NUM_PARALLEL=1, keep_alive) that must be set where ollama serve runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Give the agent recall of things said beyond the verbatim window, without
breaking the RAM-only philosophy — nothing is persisted to disk.
- MemoryIndex: a capped, in-memory pool of embedded messages with pure-Python
cosine search (no numpy). Retains far more than the rolling transcript so old
lines can be surfaced on demand; oldest evicted past the cap to bound RAM.
- OllamaEmbedder: local embeddings via nomic-embed-text, on by default and
independent of the chat provider (reuses the Ollama host when chat is Ollama).
- Bridge: captured room messages (live + backfilled) are embedded on a
background worker so a slow embedder can't stall frame draining. On a /ai
question the agent retrieves top-k relevant lines, drops weak (<min_score) and
windowed-duplicate hits, and prepends them as a clearly-fenced "recalled
context" preamble — kept at user role, never elevated to system, so untrusted
room text informs without instructing. Falls back to recency-only if the
embedder is unreachable.
- CLI: --no-rag, --embed-model, --embed-host, --rag-top-k.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Roadmap for deepening the /ai agent's conversational context while keeping
the RAM-only philosophy, plus Ollama latency wins. Marks Tier 1 (backfill,
token-budget window) and the perf tuning as in-scope now; RAG and in-RAM
compaction staged next. Grounded in public Anthropic docs, not leaked source.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bump from 960px/12fps to 1280px/15fps with floyd-steinberg dithering
for crisper, retina-legible terminal text — 7.4MB, under GitHub's 10MB
inline-render limit. Exceeds the upstream example.gif (800px/15fps).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4.7MB looping GIF rendered from the latest demo capture (alice+bob
sharing a multipass box: summon, drive, per-user sudo).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Make connecting any model a config step, not a code change:
- models.toml named profiles (api_key_env names an env var, never the key)
- providers gain available_models(); add preflight + --list-models/--check
- /ai list and /ai models in-room; client probes local Ollama for
/ai models when no agent is running, and /ai list hints to summon one
- docs/providers.md provider guide + examples/echo_provider.py
- README: command table, AI section, layout updated
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Begin the coven evolution of cmd-chat (see docs/spec-collaborative-sandbox.md):
a Rust/ratatui client for the unchanged Python Sanic server, plus the
multi-user + zero-knowledge groundwork.
P0 — crypto parity (the spec's #1 risk), proven three ways:
- Hand-rolled SRP-6a (NG_2048, SHA-256, rfc5054 padding) matching pysrp
byte-for-byte, incl. the fixed b"chat" SRP identity and minimal-vs-256B
width quirks. Golden-vector unit test + offline selftest.
- Live handshake against the running server (H_AMK verified).
- Cross-language E2E: Python client decrypts a Rust-encrypted Fernet message.
P2 — multi-user coven (server):
- CMD_CHAT_MAX_USERS capacity cap (default 4, infra-for-more).
- Authoritative roster + user_joined broadcasts.
- Free the slot/username on ws disconnect (was held until 1h stale sweep).
Also: fix requirements.txt (was UTF-16, unparseable by pip).
coven/ : Rust crate (crypto.rs proven; main.rs spike CLI: selftest/handshake/srpm)
docs/ : full feature spec for the 6 requested features.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
leetcrypt