docs: extend README demo with AI mode (flag browser + NL2SH + compare)

Re-records the home-screen demo at the readable 92x30 geometry and appends
the AI feature: English->bash one-liner generation and side-by-side model
A/B, so the README clip now showcases the full feature set in one take.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md

@@ -11,9 +11,16 @@ descriptions, and jump back to the command list instantly.
 Video (MP4): [`demo/super-man-demo.mp4`](demo/super-man-demo.mp4) ·
 asciinema cast: [`demo/super-man-demo.cast`](demo/super-man-demo.cast)
 
-The clip shows opening an entry (`grep`), browsing its flags, then pressing
-**Esc** to return straight to the home screen — no intermediary step — and
-repeating with `tar`.
+The clip fuzzy-filters to `chmod`, opens its flag browser to scroll the
+options with live man-page descriptions, then **Esc** back to the home
+screen — and finishes with AI mode: turning plain English into a bash
+one-liner and A/B-ing two local models side by side.
+
+### Themes
+
+super-man inherits your terminal's colors, so it looks at home in any palette:
+
+![super-man across dracula, monokai, nord and solarized-dark themes](demo/super-man-themes.png)
 
 ## Usage
 
@@ -28,9 +35,44 @@ super-man ask "find large files"        # ask a question
 super-man task "monitor cpu usage"      # describe what you want to do
 super-man category files                # browse by category (files/text/network/system)
 super-man explain "tar -czf"            # explain what a command does
-super-man ai "count lines in js files"  # AI-powered NL2SH model (advanced)
+super-man ai "count lines in js files"  # AI bash one-liner via local LLM (advanced)
 ```
 
+### AI mode (local LLM via ollama)
+
+![super-man ai — natural language to bash, fully local](demo/super-man-ai.gif)
+
+`ai` generates a bash one-liner from plain English using a local [ollama](https://ollama.com)
+model. The default is `westenfelder/NL2SH`, a model purpose-built for
+natural-language-to-shell — in our 25-query benchmark it was the most accurate, the
+only one that handled hard multi-stage tasks. The model is configurable, so you can
+still A/B other local LLMs:
+
+```bash
+super-man ai "find python files modified last week"          # default model (NL2SH)
+super-man ai -q "list the 5 largest files"                   # quiet: print ONLY the command
+super-man ai --model qwen2.5-coder:1.5b "count lines in js"  # override model
+super-man ai --compare "list the 5 largest files"            # run several models side by side
+```
+
+The `-q` / `--quiet` / `--raw` flag prints only the generated command to stdout (errors
+go to stderr), so it composes in scripts and pipelines:
+
+```bash
+cmd=$(super-man ai -q "compress the logs folder into a tarball")
+echo "$cmd"            # tar -czf logs.tar.gz logs/
+```
+
+| Env var | Purpose | Default |
+|---------|---------|---------|
+| `SUPERMAN_AI_MODEL` | Default model for `ai` | `westenfelder/NL2SH` |
+| `SUPERMAN_AI_COMPARE_MODELS` | Space-separated list for `--compare` | `westenfelder/NL2SH qwen2.5-coder:1.5b` |
+
+The purpose-built `westenfelder/NL2SH` model receives the raw query; general coder
+models (e.g. `qwen2.5-coder`) are prompt-wrapped to emit a single bare command. Pull
+either with `ollama pull <model>`. See the
+[benchmark write-up](tests/performance/) for the full model comparison.
+
 `Esc` in the flag browser returns to the command list; `Esc` on the home screen
 exits.
 

super-man.sh

@@ -41,7 +41,7 @@ show_help() {
   echo -e "  ${CYAN}Natural Language Queries:${NC}"
   echo -e "    ${GREEN}ask${NC} \"query\"           Ask in natural language (database)"
   echo -e "    ${GREEN}task${NC} \"description\"    Describe what you want to do (database)"
-  echo -e "    ${GREEN}ai${NC} \"complex query\"    AI-powered NL2SH model (advanced) 🤖"
+  echo -e "    ${GREEN}ai${NC} \"complex query\"    Text-to-bash one-liner, fully local (advanced) 🤖"
   echo ""
   echo -e "  ${CYAN}Browse & Explore:${NC}"
   echo -e "    ${GREEN}category${NC} NAME          Browse by category (files/text/network/system)"
@@ -76,16 +76,21 @@ show_help() {
   echo -e "  ${GREEN}✓${NC} jq (for natural language queries)"
   echo -e "  ${GREEN}✓${NC} fzf (for interactive mode only)"
   echo ""
-  echo -e "${BOLD}${YELLOW}AI MODE${NC} 🤖"
-  echo -e "  ${GREEN}✓ NL2SH model detected!${NC} Use AI for complex queries:"
+  echo -e "${BOLD}${YELLOW}AI MODE${NC} 🤖 (text-to-bash, fully local via ollama)"
   echo -e "    ${GREEN}super-man ai \"find python files modified last week\"${NC}"
-  echo -e "    ${GREEN}super-man ai \"count lines in all js files\"${NC}"
+  echo -e "    ${GREEN}super-man ai -q \"count lines in all js files\"${NC}      # print ONLY the command"
+  echo -e "    ${GREEN}super-man ai --compare \"list the 5 largest files\"${NC}  # A/B models"
   echo ""
-  echo "  Other models available: llama3.2, qwen2.5"
-  echo "  To switch models, edit mode_ai() function"
+  echo -e "  ${DIM}-q/--raw prints just the command — pipeable:${NC}"
+  echo -e "    ${GREEN}\$(super-man ai -q \"...\")${NC}   or   ${GREEN}super-man ai -q \"...\" | tee cmd.sh${NC}"
+  echo ""
+  echo -e "  Default model: ${CYAN}${SUPERMAN_AI_MODEL}${NC}"
+  echo "  Configure via env vars:"
+  echo "    SUPERMAN_AI_MODEL           default model (e.g. qwen2.5-coder:1.5b for speed)"
+  echo "    SUPERMAN_AI_COMPARE_MODELS  space-separated list for --compare"
   echo ""
   echo -e "${YELLOW}Database:${NC} $DB_FILE"
-  echo -e "${YELLOW}Version:${NC} 2.1.0 (250+ commands, AI-powered)"
+  echo -e "${YELLOW}Version:${NC} 2.3.0 (250+ commands, local text-to-bash AI)"
   echo ""
 }
 
@@ -308,41 +313,165 @@ mode_category() {
 }
 
 # AI mode - use NL2SH model for complex queries
+# Generate bash one-liners from natural language via a local ollama model.
+# The model is configurable so we can A/B different local LLMs:
+#   SUPERMAN_AI_MODEL          default model (default: qwen2.5-coder:1.5b)
+#   SUPERMAN_AI_COMPARE_MODELS space-separated list used by `ai --compare`
+# Override per call with `--model NAME`; run every comparison model side by
+# side with `--compare`. NL2SH is purpose-built for NL->shell so it receives
+# the raw query; general coder models get an instruction wrapper plus markdown
+# fence stripping so they too emit a single bare command.
+SUPERMAN_AI_MODEL="${SUPERMAN_AI_MODEL:-westenfelder/NL2SH}"
+SUPERMAN_AI_COMPARE_MODELS="${SUPERMAN_AI_COMPARE_MODELS:-westenfelder/NL2SH qwen2.5-coder:1.5b}"
+
+# Build a model-appropriate prompt for a query.
+ai_build_prompt() {
+  local model="$1" query="$2"
+  case "$model" in
+    *NL2SH*|*nl2sh*)
+      printf '%s' "$query"
+      ;;
+    *)
+      printf 'You are a bash expert. Output ONLY a single bash one-liner that accomplishes the task. No explanation, no comments, no markdown code fences. Task: %s' "$query"
+      ;;
+  esac
+}
+
+# Strip markdown fences, surrounding blank lines, and a single pair of inline
+# backticks wrapping the whole line (coder models often emit `cmd`). Only a
+# matching leading+trailing pair is removed, so command substitution like
+# `date` inside a longer command is left untouched.
+ai_clean_response() {
+  awk '/^[[:space:]]*```/ { next } { print }' \
+    | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
+    | sed -E 's/^`(.*)`$/\1/' \
+    | sed '/^$/d'
+}
+
+# True if an ollama model is pulled locally.
+ai_model_installed() {
+  ollama list 2>/dev/null | grep -q "$1"
+}
+
+# Generate a cleaned command from one model for a query.
+ai_generate() {
+  local model="$1" query="$2"
+  ollama run "$model" "$(ai_build_prompt "$model" "$query")" 2>/dev/null | ai_clean_response
+}
+
+# Print the man/help NAME line for the base command of a generated one-liner.
+ai_explain_base() {
+  local base_cmd
+  base_cmd=$(echo "$1" | awk '{print $1}')
+  if type "$base_cmd" 2>/dev/null | grep -q "shell builtin"; then
+    echo -e "${YELLOW}Command Info:${NC}"
+    help "$base_cmd" 2>/dev/null | head -3
+    echo ""
+  elif command -v "$base_cmd" &> /dev/null; then
+    echo -e "${YELLOW}Command Info:${NC}"
+    man "$base_cmd" 2>/dev/null | col -b | grep -A 2 "^NAME" | tail -2
+    echo ""
+  fi
+}
+
 mode_ai() {
-  local user_query="$1"
+  local compare=false
+  local quiet=false
+  local model="$SUPERMAN_AI_MODEL"
+  local args=()
+  while [ $# -gt 0 ]; do
+    case "$1" in
+      --compare|-c) compare=true ;;
+      --quiet|--raw|-q) quiet=true ;;
+      --model|-m) shift; model="$1" ;;
+      *) args+=("$1") ;;
+    esac
+    shift
+  done
+  local user_query="${args[*]}"
 
   if [ -z "$user_query" ]; then
-    echo -e "${RED}Error: No query provided${NC}"
-    echo "Usage: super-man ai \"your complex query\""
-    echo ""
-    echo "Example:"
-    echo "  super-man ai \"find all python files modified in last week\""
+    echo -e "${RED}Error: No query provided${NC}" >&2
+    echo "Usage: super-man ai [--model NAME] [--quiet|--compare] \"your query\"" >&2
+    echo "" >&2
+    echo "Examples:" >&2
+    echo "  super-man ai \"find all python files modified in last week\"" >&2
+    echo "  super-man ai -q \"count lines in all js files\"   # prints only the command" >&2
+    echo "  super-man ai --compare \"list the 5 largest files\"" >&2
     return 1
   fi
 
   # Check if ollama is available
   if ! command -v ollama &> /dev/null; then
-    echo -e "${RED}Error: Ollama is not installed${NC}"
-    echo ""
-    echo "Install Ollama:"
-    echo "  curl -fsSL https://ollama.com/install.sh | sh"
-    echo ""
-    echo "For now, try: super-man ask \"$user_query\""
+    echo -e "${RED}Error: Ollama is not installed${NC}" >&2
+    echo "" >&2
+    echo "Install Ollama:" >&2
+    echo "  curl -fsSL https://ollama.com/install.sh | sh" >&2
+    echo "" >&2
+    echo "For now, try: super-man ask \"$user_query\"" >&2
     return 1
   fi
 
-  # Check if NL2SH model is available
-  if ! ollama list | grep -q "westenfelder/NL2SH"; then
-    echo -e "${YELLOW}NL2SH model not found. Available models:${NC}"
+  # ── Quiet/raw mode: print ONLY the generated command (pipeable, scriptable) ─
+  if $quiet; then
+    if ! ai_model_installed "$model"; then
+      echo "Error: model '$model' not installed. Pull with: ollama pull $model" >&2
+      return 1
+    fi
+    local out
+    out=$(ai_generate "$model" "$user_query")
+    if [ -z "$out" ]; then
+      echo "Error: no response from $model" >&2
+      return 1
+    fi
+    printf '%s\n' "$out"
+    return 0
+  fi
+
+  # ── Compare mode: run every configured model side by side ──────────────────
+  if $compare; then
+    echo ""
+    echo -e "${CYAN}🤖 AI Compare${NC}"
+    echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
+    echo -e "${YELLOW}Query:${NC} $user_query"
+    echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
+    local m
+    for m in $SUPERMAN_AI_COMPARE_MODELS; do
+      echo ""
+      echo -e "${BOLD}${MAGENTA}▸ $m${NC}"
+      if ! ai_model_installed "$m"; then
+        echo -e "  ${DIM}not installed — pull with: ollama pull $m${NC}"
+        continue
+      fi
+      local start end elapsed out
+      start=$(date +%s.%N)
+      out=$(ai_generate "$m" "$user_query")
+      end=$(date +%s.%N)
+      elapsed=$(awk "BEGIN{printf \"%.1f\", $end-$start}")
+      if [ -z "$out" ]; then
+        echo -e "  ${RED}(no response)${NC}  ${DIM}(${elapsed}s)${NC}"
+      else
+        echo -e "  ${GREEN}$out${NC}  ${DIM}(${elapsed}s)${NC}"
+      fi
+    done
+    echo ""
+    echo -e "${DIM}Note: Always review AI-generated commands before running them!${NC}"
+    echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
+    return 0
+  fi
+
+  # ── Single-model mode ──────────────────────────────────────────────────────
+  if ! ai_model_installed "$model"; then
+    echo -e "${YELLOW}Model '$model' not found. Installed models:${NC}"
     ollama list
     echo ""
-    echo "Pull NL2SH model with:"
-    echo "  ollama pull westenfelder/NL2SH"
+    echo "Pull it with:    ollama pull $model"
+    echo "Or pick another: super-man ai --model NAME \"$user_query\""
     return 1
   fi
 
   echo ""
-  echo -e "${CYAN}🤖 AI Assistant - Using NL2SH Model${NC}"
+  echo -e "${CYAN}🤖 AI Assistant — $model${NC}"
   echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
   echo -e "${YELLOW}Query:${NC} $user_query"
   echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
@@ -350,8 +479,7 @@ mode_ai() {
   echo -e "${DIM}Generating bash command...${NC}"
   echo ""
 
-  # Query the NL2SH model
-  local response=$(ollama run westenfelder/NL2SH "$user_query" 2>/dev/null)
+  local response=$(ai_generate "$model" "$user_query")
 
   if [ -z "$response" ]; then
     echo -e "${RED}Error: No response from AI model${NC}"
@@ -365,19 +493,7 @@ mode_ai() {
   echo -e "  ${MAGENTA}$response${NC}"
   echo ""
 
-  # Extract the base command for explanation
-  local base_cmd=$(echo "$response" | awk '{print $1}')
-
-  # Try to provide explanation
-  if command -v "$base_cmd" &> /dev/null || type "$base_cmd" 2>/dev/null | grep -q "shell builtin"; then
-    echo -e "${YELLOW}Command Info:${NC}"
-    if type "$base_cmd" 2>/dev/null | grep -q "shell builtin"; then
-      help "$base_cmd" 2>/dev/null | head -3
-    else
-      man "$base_cmd" 2>/dev/null | col -b | grep -A 2 "^NAME" | tail -2
-    fi
-    echo ""
-  fi
+  ai_explain_base "$response"
 
   echo -e "${DIM}Note: Always review AI-generated commands before running them!${NC}"
   echo ""
@@ -1250,8 +1366,9 @@ else
       QUERY="$2"
       ;;
     ai)
-      MODE="ai"
-      QUERY="$2"
+      shift
+      mode_ai "$@"
+      exit $?
       ;;
     -*)
       echo "Unknown option: $1"
@@ -1280,10 +1397,6 @@ if [ -n "$MODE" ]; then
       mode_category "$QUERY"
       exit $?
       ;;
-    ai)
-      mode_ai "$QUERY"
-      exit $?
-      ;;
   esac
 fi
 

tests/performance/ai-benchmark.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+# AI mode benchmark harness for super-man.
+# Runs a fixed query suite against each model and records latency + the
+# generated command to a pipe-delimited CSV for analysis/reporting.
+#
+# Usage: ./ai-benchmark.sh [model1 model2 ...]
+# Default models: qwen2.5-coder:1.5b qwen2.5:3b westenfelder/NL2SH
+#
+# Output: tests/performance/ai-results.csv  (model|id|category|difficulty|query|latency_s|command)
+
+set -u
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SUPERMAN="$SCRIPT_DIR/../../super-man.sh"
+OUT="$SCRIPT_DIR/ai-results.csv"
+
+MODELS=("$@")
+if [ ${#MODELS[@]} -eq 0 ]; then
+  MODELS=("qwen2.5-coder:1.5b" "qwen2.5:3b" "westenfelder/NL2SH")
+fi
+
+# Suite: id|category|difficulty|query
+SUITE=(
+  "q01|files|easy|list the 5 largest files in the current directory"
+  "q02|files|easy|show hidden files in long format"
+  "q03|files|medium|find files larger than 100MB under the home directory"
+  "q04|files|medium|recursively change permissions of all directories to 755"
+  "q05|files|hard|find duplicate files by content in the current tree"
+  "q06|text|easy|count the number of lines in all python files"
+  "q07|text|medium|replace all tabs with two spaces in config.txt in place"
+  "q08|text|medium|print the 10 most common words in access.log"
+  "q09|text|hard|sum the numbers in the third column of data.csv"
+  "q10|search|easy|recursively search for the word TODO in all js files"
+  "q11|search|medium|find files modified in the last 24 hours"
+  "q12|search|hard|find python files that import requests but not os"
+  "q13|process|easy|show the top 10 processes by memory usage"
+  "q14|process|medium|kill the process listening on port 8080"
+  "q15|process|hard|show the total cpu time used by all chrome processes"
+  "q16|network|easy|download a file from a url and save it as out.bin"
+  "q17|network|medium|list all open listening tcp ports with the owning process"
+  "q18|network|hard|test whether port 443 is open on example.com"
+  "q19|archive|easy|compress the folder logs into a gzip tarball"
+  "q20|archive|medium|extract only txt files from archive.tar.gz"
+  "q21|system|easy|show disk usage of the current directory in human readable form"
+  "q22|system|medium|show the 5 largest directories under /var"
+  "q23|git|easy|show the last 5 git commits in one line each"
+  "q24|git|medium|list files changed in the last commit"
+  "q25|git|hard|find the commit that introduced the string getUserById"
+)
+
+strip() { sed -E 's/\x1b\[[0-9;]*m//g'; }
+
+echo "model|id|category|difficulty|query|latency_s|command" > "$OUT"
+
+for m in "${MODELS[@]}"; do
+  echo ">> Warming up $m ..."
+  ollama run "$m" "echo hi" >/dev/null 2>&1
+  echo ">> Benchmarking $m (${#SUITE[@]} queries)"
+  for row in "${SUITE[@]}"; do
+    IFS='|' read -r id cat diff q <<< "$row"
+    start=$(date +%s.%N)
+    cmd=$("$SUPERMAN" ai --model "$m" "$q" 2>&1 | strip \
+          | awk '/Generated Command:/{getline; gsub(/^[[:space:]]+/,""); print; exit}')
+    end=$(date +%s.%N)
+    el=$(awk "BEGIN{printf \"%.2f\", $end-$start}")
+    # squash any embedded pipes/newlines in the captured command for CSV safety
+    cmd=$(printf '%s' "$cmd" | tr '\n' ' ' | sed 's/|/¦/g')
+    printf "%s|%s|%s|%s|%s|%s|%s\n" "$m" "$id" "$cat" "$diff" "$q" "$el" "$cmd" >> "$OUT"
+    printf "  [%s] %-5s %5ss  %s\n" "$m" "$id" "$el" "$cmd"
+  done
+done
+
+echo ""
+echo "Done -> $OUT"

tests/performance/ai-results.csv

@@ -0,0 +1,76 @@
+model|id|category|difficulty|query|latency_s|command
+qwen2.5-coder:1.5b|q01|files|easy|list the 5 largest files in the current directory|9.42|ls -lh ¦ tail -n 5
+qwen2.5-coder:1.5b|q02|files|easy|show hidden files in long format|5.12|ls -al
+qwen2.5-coder:1.5b|q03|files|medium|find files larger than 100MB under the home directory|17.79|find ~ -type f -size +100M -exec du {} \; ¦ grep -B1 "100M" ¦¦ echo "No files found larger than 100MB."
+qwen2.5-coder:1.5b|q04|files|medium|recursively change permissions of all directories to 755|8.63|find . -type d -exec chmod 755 {} +
+qwen2.5-coder:1.5b|q05|files|hard|find duplicate files by content in the current tree|14.29|find . -type f -exec md5sum {} + ¦ sort -k1,1 ¦ uniq -d ¦ xargs --no-run-if-empty md5sum -c
+qwen2.5-coder:1.5b|q06|text|easy|count the number of lines in all python files|3.03|find . -name "*.py" ¦ xargs wc -l
+qwen2.5-coder:1.5b|q07|text|medium|replace all tabs with two spaces in config.txt in place|2.82|sed 's/	/  /g' config.txt > temp && mv temp config.txt
+qwen2.5-coder:1.5b|q08|text|medium|print the 10 most common words in access.log|4.52|awk '{print $7}' access.log ¦ tr -cd '[:alnum:]' ¦ sort ¦ uniq -c ¦ sort -nr ¦ head -n 10
+qwen2.5-coder:1.5b|q09|text|hard|sum the numbers in the third column of data.csv|2.23|awk 'NR>2{print $3}' data.csv ¦ bc
+qwen2.5-coder:1.5b|q10|search|easy|recursively search for the word TODO in all js files|3.43|find . -name "*.js" -exec grep -H "TODO" {} \;
+qwen2.5-coder:1.5b|q11|search|medium|find files modified in the last 24 hours|1.87|find . -type f -mtime -1
+qwen2.5-coder:1.5b|q12|search|hard|find python files that import requests but not os|4.06|find . -name '*.py' -exec grep -E 'import requests¦from requests import.*' {} \; -not -path '*/os.py'
+qwen2.5-coder:1.5b|q13|process|easy|show the top 10 processes by memory usage|2.50|ps aux ¦ sort -nr -k4 ¦ head -n10
+qwen2.5-coder:1.5b|q14|process|medium|kill the process listening on port 8080|2.82|kill -9 $(ss -tuln ¦ grep :8080 ¦ awk '{print $7}')
+qwen2.5-coder:1.5b|q15|process|hard|show the total cpu time used by all chrome processes|3.39|ps -e -o pcpu= --no-headers ¦ sort -hr ¦ head -n 10
+qwen2.5-coder:1.5b|q16|network|easy|download a file from a url and save it as out.bin|2.51|curl -O https://example.com/out.bin
+qwen2.5-coder:1.5b|q17|network|medium|list all open listening tcp ports with the owning process|9.46|netstat -tuln ¦ awk '$5 ~ /^/ {print $1, $3}'
+qwen2.5-coder:1.5b|q18|network|hard|test whether port 443 is open on example.com|4.61|echo -e "Checking if port 443 is open on www.example.com..."
+qwen2.5-coder:1.5b|q19|archive|easy|compress the folder logs into a gzip tarball|1.74|tar czvf logs.tar.gz logs/
+qwen2.5-coder:1.5b|q20|archive|medium|extract only txt files from archive.tar.gz|5.12|tar -xzvf archive.tar.gz -T ¦ grep "\.(txt)$"
+qwen2.5-coder:1.5b|q21|system|easy|show disk usage of the current directory in human readable form|1.95|du -h
+qwen2.5-coder:1.5b|q22|system|medium|show the 5 largest directories under /var|2.16|du -lh ¦ sort -rh ¦ head -n 5
+qwen2.5-coder:1.5b|q23|git|easy|show the last 5 git commits in one line each|1.74|git log -n 5
+qwen2.5-coder:1.5b|q24|git|medium|list files changed in the last commit|1.96|git diff --name-only HEAD~1
+qwen2.5-coder:1.5b|q25|git|hard|find the commit that introduced the string getUserById|2.59|git log --diff-filter=A --grep 'getUserById' ¦ head -n 1
+qwen2.5:3b|q01|files|easy|list the 5 largest files in the current directory|5.13|ls -lSr ¦ head -n 5
+qwen2.5:3b|q02|files|easy|show hidden files in long format|1.82|ls -alh .*
+qwen2.5:3b|q03|files|medium|find files larger than 100MB under the home directory|3.34|find ~ -type f -size +100M
+qwen2.5:3b|q04|files|medium|recursively change permissions of all directories to 755|2.82|chmod -R 755 *
+qwen2.5:3b|q05|files|hard|find duplicate files by content in the current tree|7.89|find . -type f -exec file --mime-type {} + ¦ awk '{print $NF}' ¦ sort ¦ uniq -d ¦ xargs -I {} find . -type f -name '{}'
+qwen2.5:3b|q06|text|easy|count the number of lines in all python files|3.83|find . -name "*.py" -exec grep -c . {} \;
+qwen2.5:3b|q07|text|medium|replace all tabs with two spaces in config.txt in place|3.76|sed -i 's/	/  /g' config.txt
+qwen2.5:3b|q08|text|medium|print the 10 most common words in access.log|6.96|grep -o '[^ ]\+' access.log ¦ sort ¦ uniq -c ¦ sort -nr ¦ head -n 10
+qwen2.5:3b|q09|text|hard|sum the numbers in the third column of data.csv|5.63|bash
+qwen2.5:3b|q10|search|easy|recursively search for the word TODO in all js files|15.53|find . -type f -name "*.js" -exec grep -l "TODO" {} \;
+qwen2.5:3b|q11|search|medium|find files modified in the last 24 hours|16.56|find . -type f -mtime -1 -exec ls -l {} \;
+qwen2.5:3b|q12|search|hard|find python files that import requests but not os|18.96|find . -name "*.py" -exec pygrep -l "import requests" {} \; ¦ xargs -I{} grep -ls "{}" *os*
+qwen2.5:3b|q13|process|easy|show the top 10 processes by memory usage|11.50|ps -eo pid,size,%mem,comm ¦ sort -k3nr ¦ head -n 10
+qwen2.5:3b|q14|process|medium|kill the process listening on port 8080|8.46|kill $(lsof -i :8080 -t)
+qwen2.5:3b|q15|process|hard|show the total cpu time used by all chrome processes|11.70|ps -eo pid,lstart,time ¦ grep chrome ¦ awk '{total+=$8} END {print total}'
+qwen2.5:3b|q16|network|easy|download a file from a url and save it as out.bin|5.86|wget -O out.bin "url"
+qwen2.5:3b|q17|network|medium|list all open listening tcp ports with the owning process|11.28|netstat -tuln ¦ awk '{print $4}' ¦ sort ¦ uniq -c
+qwen2.5:3b|q18|network|hard|test whether port 443 is open on example.com|8.30|nc -vz example.com 443
+qwen2.5:3b|q19|archive|easy|compress the folder logs into a gzip tarball|9.95|gzip -cvf logs/*.txt ¦ cat > logs_$(date +%Y%m%d).tar.gz
+qwen2.5:3b|q20|archive|medium|extract only txt files from archive.tar.gz|6.20|tar -zxf archive.tar.gz --wildcards '*/.*'
+qwen2.5:3b|q21|system|easy|show disk usage of the current directory in human readable form|2.19|du -sh * ..
+qwen2.5:3b|q22|system|medium|show the 5 largest directories under /var|4.83|find /var -type d -printf '%b	%p
+qwen2.5:3b|q23|git|easy|show the last 5 git commits in one line each|3.59|git log -5 --oneline --format='%H %ad ¦ %s'
+qwen2.5:3b|q24|git|medium|list files changed in the last commit|3.30|git log -1 --format="%B" --abbrev-commit ¦ wc -m
+qwen2.5:3b|q25|git|hard|find the commit that introduced the string getUserById|3.45|git log -p --diff-filter=A --summary --grep='getUserById'
+westenfelder/NL2SH|q01|files|easy|list the 5 largest files in the current directory|3.96|ls -Sh ¦ head
+westenfelder/NL2SH|q02|files|easy|show hidden files in long format|3.09|ls -la .*
+westenfelder/NL2SH|q03|files|medium|find files larger than 100MB under the home directory|5.51|find ~/ -size +100M
+westenfelder/NL2SH|q04|files|medium|recursively change permissions of all directories to 755|8.50|find . -type d -print0 ¦ xargs -0 chmod 755
+westenfelder/NL2SH|q05|files|hard|find duplicate files by content in the current tree|4.09|fdupes -r .
+westenfelder/NL2SH|q06|text|easy|count the number of lines in all python files|7.06|cat $(find . -name "*.py") ¦ wc -l
+westenfelder/NL2SH|q07|text|medium|replace all tabs with two spaces in config.txt in place|7.49|sed -i 's/	/  /' config.txt
+westenfelder/NL2SH|q08|text|medium|print the 10 most common words in access.log|17.90|cut -d' ' -f4-5 access.log ¦ tr A-Z a-z ¦ awk '{print $2}'¦ sort ¦ uniq -c ¦ sort -nr ¦ head -n10;
+westenfelder/NL2SH|q09|text|hard|sum the numbers in the third column of data.csv|17.42|awk -F, 'BEGIN{srand();}{print rand() "	" $3}' <(sort -R data.csv) ¦ awk '{sum+=$2}END{print sum}'
+westenfelder/NL2SH|q10|search|easy|recursively search for the word TODO in all js files|7.72|find . -name '*.js' ¦ xargs grep 'TODO'
+westenfelder/NL2SH|q11|search|medium|find files modified in the last 24 hours|4.20|find . -mtime -1
+westenfelder/NL2SH|q12|search|hard|find python files that import requests but not os|16.41|find . -type f \( -name "*.py" \) -exec grep -q "import requests" {} \; -a -not -exec grep -q "import os" {} \; -print
+westenfelder/NL2SH|q13|process|easy|show the top 10 processes by memory usage|8.90|ps -eo pid,ppid,%mem,cmd ¦ sort -k3nr ¦ head
+westenfelder/NL2SH|q14|process|medium|kill the process listening on port 8080|5.71|fuser -k 8080/tcp
+westenfelder/NL2SH|q15|process|hard|show the total cpu time used by all chrome processes|13.11|top -b -n 1 ¦ grep Chrome ¦ awk '{total+=$9} END {print "Total CPU Time: ", total, "s"}'
+westenfelder/NL2SH|q16|network|easy|download a file from a url and save it as out.bin|6.18|curl -o out.bin http://example.com/file.bin
+westenfelder/NL2SH|q17|network|medium|list all open listening tcp ports with the owning process|7.38|lsof -i TCP -sTCP:LISTEN ¦ grep LISTEN
+westenfelder/NL2SH|q18|network|hard|test whether port 443 is open on example.com|5.87|nc -zw1 example.com 443
+westenfelder/NL2SH|q19|archive|easy|compress the folder logs into a gzip tarball|5.01|tar -czf logs.tar.gz logs/
+westenfelder/NL2SH|q20|archive|medium|extract only txt files from archive.tar.gz|6.32|tar -zxvf archive.tar.gz --wildcards '*.txt'
+westenfelder/NL2SH|q21|system|easy|show disk usage of the current directory in human readable form|3.55|du -h .
+westenfelder/NL2SH|q22|system|medium|show the 5 largest directories under /var|7.51|du -a /var ¦ sort -n -r ¦ head -n 5
+westenfelder/NL2SH|q23|git|easy|show the last 5 git commits in one line each|4.88|git log -n 5 --oneline
+westenfelder/NL2SH|q24|git|medium|list files changed in the last commit|3.85|git whatchanged -1
+westenfelder/NL2SH|q25|git|hard|find the commit that introduced the string getUserById|13.61|git rev-list --first-parent HEAD ¦ xargs git show -s --pretty=format:%H ¦ grep 'getUserById' ¦ tail -n1

tests/performance/ai-scores.csv

@@ -0,0 +1,76 @@
+model|id|difficulty|verdict
+qwen2.5-coder:1.5b|q01|easy|wrong
+qwen2.5-coder:1.5b|q02|easy|correct
+qwen2.5-coder:1.5b|q03|medium|partial
+qwen2.5-coder:1.5b|q04|medium|correct
+qwen2.5-coder:1.5b|q05|hard|wrong
+qwen2.5-coder:1.5b|q06|easy|correct
+qwen2.5-coder:1.5b|q07|medium|correct
+qwen2.5-coder:1.5b|q08|medium|wrong
+qwen2.5-coder:1.5b|q09|hard|wrong
+qwen2.5-coder:1.5b|q10|easy|correct
+qwen2.5-coder:1.5b|q11|medium|correct
+qwen2.5-coder:1.5b|q12|hard|wrong
+qwen2.5-coder:1.5b|q13|easy|correct
+qwen2.5-coder:1.5b|q14|medium|wrong
+qwen2.5-coder:1.5b|q15|hard|wrong
+qwen2.5-coder:1.5b|q16|easy|partial
+qwen2.5-coder:1.5b|q17|medium|wrong
+qwen2.5-coder:1.5b|q18|hard|wrong
+qwen2.5-coder:1.5b|q19|easy|correct
+qwen2.5-coder:1.5b|q20|medium|wrong
+qwen2.5-coder:1.5b|q21|easy|correct
+qwen2.5-coder:1.5b|q22|medium|wrong
+qwen2.5-coder:1.5b|q23|easy|partial
+qwen2.5-coder:1.5b|q24|medium|correct
+qwen2.5-coder:1.5b|q25|hard|wrong
+qwen2.5:3b|q01|easy|wrong
+qwen2.5:3b|q02|easy|partial
+qwen2.5:3b|q03|medium|correct
+qwen2.5:3b|q04|medium|wrong
+qwen2.5:3b|q05|hard|wrong
+qwen2.5:3b|q06|easy|wrong
+qwen2.5:3b|q07|medium|correct
+qwen2.5:3b|q08|medium|correct
+qwen2.5:3b|q09|hard|wrong
+qwen2.5:3b|q10|easy|partial
+qwen2.5:3b|q11|medium|correct
+qwen2.5:3b|q12|hard|wrong
+qwen2.5:3b|q13|easy|correct
+qwen2.5:3b|q14|medium|correct
+qwen2.5:3b|q15|hard|wrong
+qwen2.5:3b|q16|easy|correct
+qwen2.5:3b|q17|medium|wrong
+qwen2.5:3b|q18|hard|correct
+qwen2.5:3b|q19|easy|wrong
+qwen2.5:3b|q20|medium|wrong
+qwen2.5:3b|q21|easy|partial
+qwen2.5:3b|q22|medium|wrong
+qwen2.5:3b|q23|easy|partial
+qwen2.5:3b|q24|medium|wrong
+qwen2.5:3b|q25|hard|wrong
+westenfelder/NL2SH|q01|easy|correct
+westenfelder/NL2SH|q02|easy|partial
+westenfelder/NL2SH|q03|medium|correct
+westenfelder/NL2SH|q04|medium|correct
+westenfelder/NL2SH|q05|hard|correct
+westenfelder/NL2SH|q06|easy|correct
+westenfelder/NL2SH|q07|medium|partial
+westenfelder/NL2SH|q08|medium|wrong
+westenfelder/NL2SH|q09|hard|partial
+westenfelder/NL2SH|q10|easy|correct
+westenfelder/NL2SH|q11|medium|correct
+westenfelder/NL2SH|q12|hard|correct
+westenfelder/NL2SH|q13|easy|correct
+westenfelder/NL2SH|q14|medium|correct
+westenfelder/NL2SH|q15|hard|wrong
+westenfelder/NL2SH|q16|easy|correct
+westenfelder/NL2SH|q17|medium|correct
+westenfelder/NL2SH|q18|hard|correct
+westenfelder/NL2SH|q19|easy|correct
+westenfelder/NL2SH|q20|medium|correct
+westenfelder/NL2SH|q21|easy|correct
+westenfelder/NL2SH|q22|medium|partial
+westenfelder/NL2SH|q23|easy|correct
+westenfelder/NL2SH|q24|medium|correct
+westenfelder/NL2SH|q25|hard|wrong