giglez/docs/DATA_CLEANUP_GUIDE.md

# Data Cleanup Guide

## Overview

GigLez now supports marking and cleaning up test/mock data from the database. This allows you to populate the platform with test data for development, then easily remove it when moving to production.

## Data Source Tracking

### Data Source Types

All captures are now tagged with a `data_source` field:

- **`test`** - Test data for development (default for CLI imports)
- **`mock`** - Mock/synthetic data
- **`production`** - Real wardriving data from field captures

### How Data Sources Are Assigned

#### During Upload (Web Interface)
- Defaults to `production`
- Can be overridden in upload manifest

#### During CLI Import
```bash
# Import as test data (default)
python3 scripts/import_wardriving_data.py data/ --data-source test

# Import as mock data
python3 scripts/import_wardriving_data.py data/ --data-source mock

# Import as production data
python3 scripts/import_wardriving_data.py data/ --data-source production
```

## Cleanup Methods

### Method 1: Cleanup Script (Recommended)

**Location**: `scripts/cleanup_database.py`

#### Clean by Data Source
```bash
# Delete all test data
python3 scripts/cleanup_database.py --source test --stats

# Delete all mock data
python3 scripts/cleanup_database.py --source mock --stats

# View stats before/after with --stats flag
python3 scripts/cleanup_database.py --source test --stats
```

#### Clean by Session ID
```bash
# Delete all captures from a specific import session
python3 scripts/cleanup_database.py --session wardrive_import_15 --stats
```

#### Clean All Data
```bash
# ⚠️ WARNING: Deletes EVERYTHING
python3 scripts/cleanup_database.py --all --confirm --stats
```

### Method 2: Direct API Calls

#### Delete by Data Source
```bash
# Delete test data
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup?data_source=test"

# Response:
{
  "success": true,
  "message": "Deleted 15 captures with data_source='test'",
  "deleted_count": 15,
  "remaining_count": 5
}
```

#### Delete by Session ID
```bash
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/session/wardrive_import_15"
```

#### Delete All Data
```bash
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/all"
```

## Check Data Sources

### View Statistics
```bash
curl -s http://localhost:8000/api/v1/stats/summary | jq '{total_captures, data_sources}'
```

**Example Output**:
```json
{
  "total_captures": 35,
  "data_sources": {
    "unknown": 20,
    "test": 15
  }
}
```

### List All Captures with Data Source
```bash
curl -s http://localhost:8000/api/v1/query/captures | jq '.captures[] | {filename, data_source}'
```

## Typical Workflows

### Workflow 1: Development Testing

```bash
# 1. Import test data
python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/ --data-source test

# 2. Test features on web interface
open http://localhost:8000

# 3. Clean up test data when done
python3 scripts/cleanup_database.py --source test --stats
```

### Workflow 2: Production Deployment

```bash
# 1. Clean all test/mock data before production
python3 scripts/cleanup_database.py --source test
python3 scripts/cleanup_database.py --source mock

# 2. Import real wardriving data
python3 scripts/import_wardriving_data.py real_captures/ --data-source production

# 3. Verify only production data remains
curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
```

### Workflow 3: Complete Reset

```bash
# Delete EVERYTHING and start fresh
python3 scripts/cleanup_database.py --all --confirm --stats
```

## Migrating Existing Data

Existing captures without a `data_source` field will show as `"unknown"` in statistics.

To clean up unknown captures:
```bash
# Manually edit data/captures_simple.json and add "data_source" field
# Or delete and re-import with proper marking
python3 scripts/cleanup_database.py --all --confirm
python3 scripts/import_wardriving_data.py data/ --data-source production
```

## API Endpoints

### GET /api/v1/stats/summary
Returns database statistics including data source breakdown:
```json
{
  "total_captures": 35,
  "unique_devices": 7,
  "data_sources": {
    "test": 15,
    "production": 20
  },
  ...
}
```

### DELETE /api/v1/admin/cleanup?data_source={source}
Delete captures by data source (`test`, `mock`, or `production`)

### DELETE /api/v1/admin/cleanup/session/{session_id}
Delete all captures from a specific upload session

### DELETE /api/v1/admin/cleanup/all
Delete ALL captures (use with extreme caution!)

## Examples

### Example 1: Import and Clean Test Data

```bash
# Import 15 test captures
$ python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/
📤 Uploading 15 captures to http://localhost:8000
   Data source: test
...
✅ Import complete!

# Check stats
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
{
  "test": 15
}

# Clean up test data
$ python3 scripts/cleanup_database.py --source test --stats
BEFORE cleanup:
📊 Current Database Stats:
   Total captures: 15
   By data source:
     test: 15

✅ Deleted 15 captures with data_source='test'
   Deleted: 15
   Remaining: 0

AFTER cleanup:
📊 Current Database Stats:
   Total captures: 0
```

### Example 2: Mixed Data Sources

```bash
# Import test data
$ python3 scripts/import_wardriving_data.py test_data/ --data-source test

# Import production data
$ python3 scripts/import_wardriving_data.py real_data/ --data-source production

# Check distribution
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
{
  "test": 15,
  "production": 10
}

# Delete only test data, keep production
$ python3 scripts/cleanup_database.py --source test
✅ Deleted 15 captures with data_source='test'
   Remaining: 10
```

## Safety Features

1. **Confirmation Required**: `--all` cleanup requires `--confirm` flag
2. **Data Source Validation**: Only accepts valid source types
3. **Persistent Storage**: Changes are immediately saved to disk
4. **Statistics Reporting**: Always shows deleted/remaining counts

## Best Practices

1. **Always use `--stats`** to see before/after counts
2. **Mark test data clearly** using `--data-source test`
3. **Use sessions** for grouping related imports
4. **Backup production data** before mass deletions
5. **Clean test data regularly** to avoid confusion

## Troubleshooting

### Issue: Can't Delete Unknown Data

**Solution**: Unknown data has no `data_source` field. Use the "all" cleanup:
```bash
python3 scripts/cleanup_database.py --all --confirm
```

### Issue: Cleanup Not Persisting

**Solution**: Check that `data/captures_simple.json` is writable:
```bash
ls -la data/captures_simple.json
```

### Issue: Wrong Data Source Marked

**Solution**: Clean and re-import with correct marking:
```bash
python3 scripts/cleanup_database.py --source test
python3 scripts/import_wardriving_data.py data/ --data-source production
```

## Summary

The data cleanup system allows you to:
- ✅ Mark uploads as test/mock/production
- ✅ Clean up by data source
- ✅ Clean up by session ID
- ✅ View statistics by data source
- ✅ Safely manage test data during development

**Happy testing!** 🧪