super-man/tests/README.md

Super Man Test Suite

Comprehensive testing framework for Super Man with performance analysis and automated reporting.

Overview

This test suite provides:

• Automated testing for all operating modes
• Performance benchmarking with statistical analysis
• Interactive mode testing using expect automation
• Detailed reporting in JSON, Markdown, and HTML formats
• CI/CD integration support

Quick Start

Run All Tests

cd tests/
./run-all-tests.sh

This will:

1. Run all non-interactive mode tests
2. Run interactive mode tests (if fzf available)
3. Generate performance benchmarks
4. Create comprehensive reports

Run Specific Test Suites

# Test non-interactive modes only
./test-all-modes.sh

# Test interactive mode only
./test-interactive-mode.sh

# Analyze performance data
./analyze-performance.sh

Test Suite Structure

tests/
├── run-all-tests.sh           # Master test runner
├── test-all-modes.sh          # Non-interactive mode tests
├── test-interactive-mode.sh   # Interactive mode tests
├── test-framework.sh          # Testing utilities & framework
├── analyze-performance.sh     # Performance analysis tool
├── logs/                      # Test execution logs
│   ├── test_*.log            # Individual test logs
│   └── test-report.json      # JSON test results
├── performance/               # Performance data
│   └── benchmarks.csv        # Benchmark results
└── reports/                   # Generated reports
    ├── performance-report.md # Markdown report
    ├── performance-report.html # HTML report
    └── ci-report.txt         # CI/CD compatible report

Test Coverage

Non-Interactive Modes

Help & Informational

• --help / -h - Help display with ASCII banner
• --list / -l - List all commands

Natural Language Queries

• ask "query" - Database search (5 test cases)
• task "description" - Task search (2 test cases)

Category Browsing

• category files - File operations
• category text - Text processing
• category network - Network operations
• category system - System monitoring
• category - List all categories
• Invalid category handling

Command Explanation

• explain "command" - Command breakdown (4 test cases)

Direct Flag Lookup

• command filter - Flag search (6 test cases)
• Command without filter

AI Mode (if available)

• ai "query" - AI-powered command generation (3 test cases with 30s timeout)

Error Handling

• Invalid mode
• Empty queries
• Special characters
• Long queries

Interactive Mode Tests

Functionality Tests

1. Launch and Exit - Verify interactive mode starts
2. Command Selection - Select command and view flags
3. Filtering - Filter commands by category/keyword
4. Multiple Commands - Test ls, tar, find selections

Performance Tests

• Startup time benchmarking (5 iterations)
• Average response time analysis

Performance Benchmarks

Targets

Mode	Target	Typical
Database search	<100ms	~50ms
Category browse	<100ms	~30ms
Direct lookup	<200ms	~100ms
Help display	<100ms	~20ms
Interactive startup	<100ms	~50ms
AI mode (first)	<20s	5-15s
AI mode (cached)	<5s	1-3s

Metrics Collected

For each test:

• Execution time (milliseconds)
• Exit code (success/failure)
• Output validation (content checks)

For benchmarks (10 iterations):

• Average time
• Min/Max times
• Success rate
• Standard deviation

Reports

JSON Report (logs/test-report.json)

{
  "timestamp": "2025-01-XX XX:XX:XX",
  "summary": {
    "total": 45,
    "passed": 43,
    "failed": 0,
    "skipped": 2
  },
  "tests": [
    {
      "name": "Ask mode - find large files",
      "result": "PASSED",
      "duration_ms": 52
    }
  ]
}

Performance Report (reports/performance-report.md)

Markdown report with:

• Benchmark results table
• Performance targets comparison
• Summary statistics
• Historical trends

HTML Report (reports/performance-report.html)

Interactive HTML dashboard with:

• Visual metrics
• Color-coded results
• Responsive design
• Print-friendly layout

Requirements

Required

• bash 4.0+ - Shell interpreter
• super-man.sh - Script being tested

Optional

• jq - JSON parsing (for detailed analysis)
• fzf - Fuzzy finder (for interactive tests)
• expect - Interactive automation (auto-installed if missing)
• ollama - AI integration (for AI mode tests)
• bc - Calculator (for percentage calculations)

Installing Dependencies

# Ubuntu/Debian
sudo apt-get install -y jq fzf expect bc

# Fedora/RHEL
sudo dnf install -y jq fzf expect bc

# macOS
brew install jq fzf expect bc

Usage Examples

Basic Test Run

./run-all-tests.sh

Output:

═══════════════════════════════════════════════════════════
  Super Man Test Suite
  2025-01-XX XX:XX:XX
═══════════════════════════════════════════════════════════

▶ Testing: Help & Informational Modes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Test #1: Help display (--help)
✓ PASSED (45ms) - Output contains: "Super Man"

Test #2: List all commands (--list)
✓ PASSED (32ms) - Output contains: "Available commands"

...

═══════════════════════════════════════════════════════════
  Test Summary
═══════════════════════════════════════════════════════════

Results:
  Total Tests:  45
  Passed:       43
  Failed:       0
  Skipped:      2

Pass Rate: 95%

Run Specific Mode Tests

# Test only natural language queries
./test-all-modes.sh 2>&1 | grep -A 20 "Natural Language"

# Test only AI mode
./test-all-modes.sh 2>&1 | grep -A 10 "AI Mode"

Performance Analysis

./analyze-performance.sh

Output:

═══════════════════════════════════════════════════════════
  Super Man Performance Analysis
  2025-01-XX XX:XX:XX
═══════════════════════════════════════════════════════════

Benchmark Analysis
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Ask mode - find large files
  Timestamp: 2025-01-XX XX:XX:XX
  Average:   48ms
  Range:     42ms - 55ms
  Success:   10/10 iterations

CI/CD Integration

# Run tests and capture exit code
./run-all-tests.sh
EXIT_CODE=$?

# Check CI report
cat reports/ci-report.txt

# Exit with test result code
exit $EXIT_CODE

Continuous Integration

GitHub Actions Example

name: Test Super Man

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y jq fzf expect bc

      - name: Run tests
        run: |
          cd tests/
          ./run-all-tests.sh

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-results
          path: tests/reports/

Gitea CI Example

# .gitea/workflows/test.yml
name: Test Suite

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Run tests
        run: |
          cd tests/
          ./run-all-tests.sh

      - name: Publish results
        if: always()
        run: |
          echo "Test results:"
          cat tests/reports/ci-report.txt

Troubleshooting

Tests Fail to Run

Problem: bash: ./run-all-tests.sh: Permission denied

Solution:

chmod +x tests/*.sh

Interactive Tests Fail

Problem: Interactive tests skip or fail

Reasons:

• fzf not installed
• expect not installed
• Running in non-interactive environment

Solution:

# Install dependencies
sudo apt-get install -y fzf expect

# Check if interactive shell
echo $-  # Should contain 'i'

AI Tests Skipped

Problem: AI mode tests are skipped

Reasons:

• Ollama not installed
• NL2SH model not available

Solution:

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull NL2SH model
ollama pull westenfelder/NL2SH

Performance Benchmarks Too Slow

Problem: Tests take too long

Solutions:

• Reduce iteration count in benchmark calls
• Skip AI tests (they take 5-15s per query)
• Run specific test suites instead of all tests

Edit iterations:

# In test scripts, change:
run_benchmark "test name" "command" 10
# to:
run_benchmark "test name" "command" 3

JSON Report Not Generated

Problem: jq not available

Solution:

sudo apt-get install -y jq

Advanced Usage

Custom Test Configuration

Create tests/config.sh:

# Test configuration
export TEST_TIMEOUT=30
export BENCHMARK_ITERATIONS=10
export SKIP_AI_TESTS=false
export SKIP_INTERACTIVE_TESTS=false

Add Custom Tests

1. Create test file:

cd tests/
cat > test-custom.sh << 'EOF'
#!/bin/bash
source ./test-framework.sh

print_header
print_section "Custom Tests"

run_test_with_output_check \
    "My custom test" \
    "../super-man.sh ask 'my query'" \
    "expected output"

print_summary
EOF

chmod +x test-custom.sh

2. Run custom test:

./test-custom.sh

Parallel Test Execution

# Run multiple test suites in parallel
./test-all-modes.sh &
./test-interactive-mode.sh &
wait

# Analyze results
./analyze-performance.sh

Export Test Data

# Export to CSV
./run-all-tests.sh
cat performance/benchmarks.csv | column -t -s '|'

# Export to JSON
cat logs/test-report.json | jq '.'

Test Framework API

Core Functions

run_test(name, command, expected_exit_code, timeout)

Run test with basic exit code checking.

run_test "Test name" "command to run" 0 10

run_test_with_output_check(name, command, expected_string, timeout)

Run test and verify output contains expected string.

run_test_with_output_check \
    "Help test" \
    "./super-man.sh --help" \
    "Super Man"

run_benchmark(name, command, iterations)

Run performance benchmark with statistics.

run_benchmark "Ask mode" "./super-man.sh ask 'test'" 10

skip_test(name, reason)

Skip test with reason.

skip_test "AI mode" "Ollama not available"

Contributing Tests

Adding New Test Cases

1. Identify test category (help, ask, explain, etc.)
2. Add test to appropriate section in test-all-modes.sh
3. Follow naming convention: Mode - specific test case
4. Include output validation
5. Document expected behavior

Example:

run_test_with_output_check \
    "Ask mode - new test case" \
    "$SCRIPT_PATH ask 'new query'" \
    "expected output keyword"

Testing Checklist

Before submitting:

• Test passes locally
• Test has unique, descriptive name
• Output validation is specific
• Timeout is appropriate (10s default, 30s for AI)
• Test doesn't require manual interaction
• Dependencies are documented

Maintenance

Cleaning Test Data

# Remove all logs and reports
rm -rf tests/logs/* tests/performance/* tests/reports/*

# Run fresh test
./run-all-tests.sh

Archiving Test Results

# Archive with timestamp
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
tar -czf test-results-$TIMESTAMP.tar.gz \
    tests/logs tests/performance tests/reports

# Move to archive directory
mkdir -p archive/
mv test-results-$TIMESTAMP.tar.gz archive/

Performance Tuning

Speed Up Tests

1. Skip slow tests:

   # Comment out AI tests if not needed
   # Comment out benchmark sections
   

2. Reduce iterations:

   # Change from 10 to 3 iterations
   run_benchmark "test" "command" 3
   

3. Parallel execution:

   ./test-all-modes.sh & ./test-interactive-mode.sh & wait
   

Optimize for CI/CD

# Fast CI mode
export TEST_TIMEOUT=5
export BENCHMARK_ITERATIONS=3
export SKIP_AI_TESTS=true
./run-all-tests.sh

FAQ

Q: How long do tests take? A: ~2-3 minutes for all tests (5-10 minutes with AI tests)

Q: Can I run tests in parallel? A: Yes, but be careful with shared resources (log files, etc.)

Q: Do I need root access? A: No, unless installing dependencies

Q: Will tests modify my system? A: No, tests only read/execute super-man.sh

Q: Can I run tests on macOS? A: Yes, with dependencies installed via Homebrew

Q: How do I debug a failing test? A: Check tests/logs/test_N_*.log for detailed output

Version History

v1.0.0 (Current)

• Initial test framework
• 45+ test cases across all modes
• Performance benchmarking
• Interactive mode automation
• Comprehensive reporting (JSON, MD, HTML)
• CI/CD integration support

License

Same as Super Man - see main repository

Support

• Issues: Report via Gitea issue tracker
• Documentation: See main repository README
• Questions: Check FAQ above or test logs

---

Super Man Test Suite v1.0.0

Ensuring quality and performance at every commit! 🚀