giglez/POSTGRESQL_SETUP_EXPLANATION.md

# PostgreSQL Setup - Why I Cannot Complete It

## Current Situation

**PostgreSQL Status**: ✅ Installed and running
```bash
$ systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
     Active: active (exited) since Mon 2026-01-12 06:49:52 PST; 10h ago
```

**Problem**: 🔒 **I don't have sudo privileges**

---

## What Needs to Happen

To set up PostgreSQL for GigLez, we need to:

### 1. Create Database User
```sql
CREATE USER giglez_user WITH PASSWORD 'giglez_dev_password';
```

### 2. Create Database
```sql
CREATE DATABASE giglez OWNER giglez_user;
```

### 3. Enable PostGIS Extension
```sql
\c giglez
CREATE EXTENSION postgis;
```

### 4. Grant Permissions
```sql
GRANT ALL PRIVILEGES ON DATABASE giglez TO giglez_user;
```

---

## Why I Cannot Do This

### Problem: Requires `sudo` Access

**All PostgreSQL administrative tasks require**:
```bash
sudo -u postgres psql
```

**When I try**:
```bash
$ sudo -u postgres psql
sudo: a password is required
```

**Result**: ❌ Cannot execute without your password

---

## The Setup Script (Already Created)

**File**: `scripts/quick_db_setup.sh`

**What it does**:
```bash
#!/bin/bash
# 1. Check PostgreSQL is running
# 2. Use sudo to connect as postgres user
# 3. Create giglez_user with password
# 4. Create giglez database
# 5. Enable PostGIS extension
# 6. Grant privileges
# 7. Create schema (tables, indexes)
```

**Why I can't run it**: Line 28 requires sudo:
```bash
sudo -u postgres psql << 'EOF'
  CREATE USER giglez_user ...
EOF
```

---

## What YOU Need to Do

### Option 1: Run the Setup Script (Recommended)

**One command** (requires your password):
```bash
./scripts/quick_db_setup.sh
```

**What will happen**:
1. Prompt for sudo password
2. Create database user `giglez_user`
3. Create database `giglez`
4. Enable PostGIS extension
5. Create all tables (devices, signatures, captures)
6. Create indexes for performance

**Time**: ~30 seconds

---

### Option 2: Manual Setup (If script fails)

**Step-by-step commands** (you'll be prompted for password):

#### 1. Connect to PostgreSQL
```bash
sudo -u postgres psql
```

#### 2. Create User and Database
```sql
-- Create user
CREATE USER giglez_user WITH PASSWORD 'giglez_dev_password';

-- Create database
CREATE DATABASE giglez OWNER giglez_user;

-- Grant privileges
GRANT ALL PRIVILEGES ON DATABASE giglez TO giglez_user;

-- Exit
\q
```

#### 3. Enable PostGIS
```bash
sudo -u postgres psql -d giglez -c "CREATE EXTENSION IF NOT EXISTS postgis;"
```

#### 4. Create Schema
```bash
psql -U giglez_user -d giglez -h localhost << 'EOF'
-- You'll be prompted for password: giglez_dev_password

-- Devices table
CREATE TABLE IF NOT EXISTS devices (
    id SERIAL PRIMARY KEY,
    device_name VARCHAR(200),
    manufacturer VARCHAR(100),
    model VARCHAR(100),
    device_type VARCHAR(50),
    typical_frequency INTEGER,
    protocol VARCHAR(100),
    description TEXT,
    first_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_verified BOOLEAN DEFAULT FALSE
);

-- Signatures table
CREATE TABLE IF NOT EXISTS signatures (
    id SERIAL PRIMARY KEY,
    device_id INTEGER REFERENCES devices(id),
    protocol VARCHAR(100),
    frequency INTEGER,
    modulation VARCHAR(50),
    bit_pattern BYTEA,
    bit_mask BYTEA,
    timing_min INTEGER,
    timing_max INTEGER,
    raw_pattern TEXT,
    confidence_threshold FLOAT DEFAULT 0.7,
    source VARCHAR(50),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Captures table
CREATE TABLE IF NOT EXISTS captures (
    file_hash VARCHAR(64) PRIMARY KEY,
    filename VARCHAR(500),
    frequency INTEGER,
    protocol VARCHAR(100),
    latitude DECIMAL(10, 8),
    longitude DECIMAL(11, 8),
    captured_at TIMESTAMP,
    device_id INTEGER REFERENCES devices(id),
    match_confidence FLOAT,
    uploaded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Indexes
CREATE INDEX IF NOT EXISTS idx_signatures_frequency ON signatures(frequency);
CREATE INDEX IF NOT EXISTS idx_signatures_device ON signatures(device_id);
CREATE INDEX IF NOT EXISTS idx_captures_frequency ON captures(frequency);

\q
EOF
```

---

## After PostgreSQL Setup

### 1. Import Signature Data

**Import Flipper Zero signatures** (85 devices):
```bash
# Adapt SQLite script for PostgreSQL
python3 scripts/import_flipper_to_postgres.py
```

**Or use existing SQLite data**:
```bash
# Convert SQLite to PostgreSQL
sqlite3 giglez.db .dump | psql -U giglez_user -d giglez -h localhost
```

### 2. Switch to Full API

**Stop simplified server**:
```bash
# Find process
ps aux | grep main_simple
kill <PID>
```

**Start full API**:
```bash
python3 src/api/main.py
```

**Verify connection**:
```bash
curl http://localhost:8000/health
# Should show: "database": "connected"
```

---

## Why PostgreSQL vs SQLite?

### Current Situation: SQLite ✅

**File**: `giglez.db` (72 KB, 85 devices)

**Advantages**:
- ✅ No setup required
- ✅ Single file
- ✅ Fast for small datasets
- ✅ Already populated with Flipper signatures

**Limitations**:
- ❌ No geographic queries (PostGIS)
- ❌ Limited concurrency
- ❌ No spatial indexing
- ❌ Doesn't scale to millions of records

### Production Goal: PostgreSQL + PostGIS

**Advantages**:
- ✅ **PostGIS** for geographic queries (radius search, bounding box)
- ✅ **Spatial indexing** (GiST) for performance
- ✅ **Scalability** (millions of captures)
- ✅ **Concurrent writes** (multiple users uploading)
- ✅ **Advanced queries** (complex geo searches)

**Example PostGIS query**:
```sql
-- Find all captures within 10km of coordinates
SELECT * FROM captures
WHERE ST_DWithin(
    ST_MakePoint(longitude, latitude)::geography,
    ST_MakePoint(-74.0060, 40.7128)::geography,
    10000  -- 10km in meters
);
```

**This is impossible in SQLite!**

---

## Current Workaround

### Why We Built `main_simple.py`

**Purpose**: Test web interface without database dependency

**What works**:
- ✅ Web interface (HTML/CSS/JS)
- ✅ Map visualization
- ✅ Navigation
- ✅ API documentation

**What doesn't work**:
- ❌ File uploads (no backend processing)
- ❌ Device matching (no database)
- ❌ Search (no data)
- ❌ Real statistics (shows zeros)

**This is TEMPORARY** - meant only for UI/UX testing.

---

## Comparison Table

| Feature | SQLite (Current) | PostgreSQL (Needed) | Simple Mode (Testing) |
|---------|------------------|---------------------|-----------------------|
| **Setup** | ✅ Done | ⏳ Needs sudo | ✅ None |
| **Signatures** | ✅ 85 loaded | ⏳ Need import | ❌ None |
| **Geographic queries** | ❌ No PostGIS | ✅ PostGIS | ❌ None |
| **Uploads** | ⏳ Possible | ✅ Full featured | ❌ Disabled |
| **Scalability** | ⚠️ ~10K records | ✅ Millions | N/A |
| **Concurrent users** | ⚠️ Limited | ✅ Unlimited | N/A |
| **Web interface** | ✅ Works | ✅ Works | ✅ Works |

---

## Step-by-Step: What You Need to Do

### Phase 1: PostgreSQL Setup (5 minutes)

```bash
# 1. Run setup script (enter password when prompted)
cd /home/dell/coding/giglez
./scripts/quick_db_setup.sh

# Expected output:
# ✅ PostgreSQL is running
# ✅ User and database created
# ✅ PostGIS enabled
# ✅ Schema created
# ✅ Database setup complete!
```

### Phase 2: Import Signatures (2 minutes)

**Option A: From SQLite** (quick):
```bash
# Export from SQLite
sqlite3 giglez.db ".dump devices signatures" > data.sql

# Import to PostgreSQL
psql -U giglez_user -d giglez -h localhost -f data.sql
# Password: giglez_dev_password
```

**Option B: Re-import from Flipper** (fresh):
```bash
# Create PostgreSQL version of import script
python3 scripts/import_flipper_to_postgres.py
```

### Phase 3: Start Full API (1 minute)

```bash
# Stop simple server
pkill -f main_simple

# Start full API
python3 src/api/main.py

# Test
curl http://localhost:8000/health
# Should show: "database": "connected"
```

### Phase 4: Test Everything (5 minutes)

```bash
# Open browser
http://localhost:8000

# 1. Upload a .sub file
# 2. See it on the map
# 3. Search for it
# 4. View statistics
```

---

## Why This Matters

### Current State: "Hello World"
- Web interface loads
- UI/UX testable
- **But no real functionality**

### After PostgreSQL: "Production MVP"
- Upload .sub files
- Automatic device identification
- Geographic search
- Interactive map with real data
- Statistics dashboard with real numbers
- **Actual Wigle-style platform!**

---

## Security Notes

### Default Password (Development)

**Current**: `giglez_dev_password`

**WARNING**: This is in `.env.development` - fine for local testing, **NOT for production**

**For production**, change to strong password:
```bash
# Generate random password
openssl rand -base64 32

# Update .env.production
GIGLEZ_DB_PASSWORD=<strong-random-password>
```

### Connection String

**Development**:
```
postgresql://giglez_user:giglez_dev_password@localhost:5432/giglez
```

**Production**:
- Use environment variables
- Encrypt connection
- Restrict network access
- Use SSL certificates

---

## Troubleshooting

### "PostgreSQL is not running"

```bash
sudo systemctl start postgresql
sudo systemctl enable postgresql  # Start on boot
```

### "Role 'giglez_user' already exists"

```bash
# Drop and recreate
sudo -u postgres psql -c "DROP USER IF EXISTS giglez_user;"
./scripts/quick_db_setup.sh
```

### "Database 'giglez' already exists"

```bash
# Drop and recreate
sudo -u postgres psql -c "DROP DATABASE IF EXISTS giglez;"
./scripts/quick_db_setup.sh
```

### "Permission denied"

```bash
# Grant all privileges
sudo -u postgres psql -d giglez -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO giglez_user;"
```

---

## Summary

### What I Did ✅

1. ✅ Created full API (`src/api/main.py`)
2. ✅ Created simplified test version (`src/api/main_simple.py`)
3. ✅ Created setup script (`scripts/quick_db_setup.sh`)
4. ✅ Documented everything

### What I Cannot Do 🔒

1. ❌ Run `sudo` commands (need your password)
2. ❌ Create PostgreSQL user
3. ❌ Create PostgreSQL database
4. ❌ Enable PostGIS extension

### What YOU Need to Do 👤

**Single command**:
```bash
./scripts/quick_db_setup.sh
```

**Then**:
```bash
python3 src/api/main.py
```

**That's it!** 🎉

---

## Current Status

**Web Interface**: ✅ Working (simplified mode)
```
http://localhost:8000
```

**Database**: ⏳ Waiting for your setup
```bash
./scripts/quick_db_setup.sh
```

**Next Step**: Run the setup script when ready!

---

**Created**: 2026-01-12
**Status**: PostgreSQL setup documented and ready
**Action Required**: User needs to run `./scripts/quick_db_setup.sh`