giglez/docs/DATA_CLEANUP_GUIDE.md

Data Cleanup Guide

Overview

GigLez now supports marking and cleaning up test/mock data from the database. This allows you to populate the platform with test data for development, then easily remove it when moving to production.

Data Source Tracking

Data Source Types

All captures are now tagged with a data_source field:

• test - Test data for development (default for CLI imports)
• mock - Mock/synthetic data
• production - Real wardriving data from field captures

How Data Sources Are Assigned

During Upload (Web Interface)

• Defaults to production
• Can be overridden in upload manifest

During CLI Import

# Import as test data (default)
python3 scripts/import_wardriving_data.py data/ --data-source test

# Import as mock data
python3 scripts/import_wardriving_data.py data/ --data-source mock

# Import as production data
python3 scripts/import_wardriving_data.py data/ --data-source production

Cleanup Methods

Method 1: Cleanup Script (Recommended)

Location: scripts/cleanup_database.py

Clean by Data Source

# Delete all test data
python3 scripts/cleanup_database.py --source test --stats

# Delete all mock data
python3 scripts/cleanup_database.py --source mock --stats

# View stats before/after with --stats flag
python3 scripts/cleanup_database.py --source test --stats

Clean by Session ID

# Delete all captures from a specific import session
python3 scripts/cleanup_database.py --session wardrive_import_15 --stats

Clean All Data

# ⚠️ WARNING: Deletes EVERYTHING
python3 scripts/cleanup_database.py --all --confirm --stats

Method 2: Direct API Calls

Delete by Data Source

# Delete test data
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup?data_source=test"

# Response:
{
  "success": true,
  "message": "Deleted 15 captures with data_source='test'",
  "deleted_count": 15,
  "remaining_count": 5
}

Delete by Session ID

curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/session/wardrive_import_15"

Delete All Data

curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/all"

Check Data Sources

View Statistics

curl -s http://localhost:8000/api/v1/stats/summary | jq '{total_captures, data_sources}'

Example Output:

{
  "total_captures": 35,
  "data_sources": {
    "unknown": 20,
    "test": 15
  }
}

List All Captures with Data Source

curl -s http://localhost:8000/api/v1/query/captures | jq '.captures[] | {filename, data_source}'

Typical Workflows

Workflow 1: Development Testing

# 1. Import test data
python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/ --data-source test

# 2. Test features on web interface
open http://localhost:8000

# 3. Clean up test data when done
python3 scripts/cleanup_database.py --source test --stats

Workflow 2: Production Deployment

# 1. Clean all test/mock data before production
python3 scripts/cleanup_database.py --source test
python3 scripts/cleanup_database.py --source mock

# 2. Import real wardriving data
python3 scripts/import_wardriving_data.py real_captures/ --data-source production

# 3. Verify only production data remains
curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'

Workflow 3: Complete Reset

# Delete EVERYTHING and start fresh
python3 scripts/cleanup_database.py --all --confirm --stats

Migrating Existing Data

Existing captures without a data_source field will show as "unknown" in statistics.

To clean up unknown captures:

# Manually edit data/captures_simple.json and add "data_source" field
# Or delete and re-import with proper marking
python3 scripts/cleanup_database.py --all --confirm
python3 scripts/import_wardriving_data.py data/ --data-source production

API Endpoints

GET /api/v1/stats/summary

Returns database statistics including data source breakdown:

{
  "total_captures": 35,
  "unique_devices": 7,
  "data_sources": {
    "test": 15,
    "production": 20
  },
  ...
}

DELETE /api/v1/admin/cleanup?data_source={source}

Delete captures by data source (test, mock, or production)

DELETE /api/v1/admin/cleanup/session/{session_id}

Delete all captures from a specific upload session

DELETE /api/v1/admin/cleanup/all

Delete ALL captures (use with extreme caution!)

Examples

Example 1: Import and Clean Test Data

# Import 15 test captures
$ python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/
📤 Uploading 15 captures to http://localhost:8000
   Data source: test
...
✅ Import complete!

# Check stats
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
{
  "test": 15
}

# Clean up test data
$ python3 scripts/cleanup_database.py --source test --stats
BEFORE cleanup:
📊 Current Database Stats:
   Total captures: 15
   By data source:
     test: 15

✅ Deleted 15 captures with data_source='test'
   Deleted: 15
   Remaining: 0

AFTER cleanup:
📊 Current Database Stats:
   Total captures: 0

Example 2: Mixed Data Sources

# Import test data
$ python3 scripts/import_wardriving_data.py test_data/ --data-source test

# Import production data
$ python3 scripts/import_wardriving_data.py real_data/ --data-source production

# Check distribution
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
{
  "test": 15,
  "production": 10
}

# Delete only test data, keep production
$ python3 scripts/cleanup_database.py --source test
✅ Deleted 15 captures with data_source='test'
   Remaining: 10

Safety Features

1. Confirmation Required: --all cleanup requires --confirm flag
2. Data Source Validation: Only accepts valid source types
3. Persistent Storage: Changes are immediately saved to disk
4. Statistics Reporting: Always shows deleted/remaining counts

Best Practices

1. Always use --stats to see before/after counts
2. Mark test data clearly using --data-source test
3. Use sessions for grouping related imports
4. Backup production data before mass deletions
5. Clean test data regularly to avoid confusion

Troubleshooting

Issue: Can't Delete Unknown Data

Solution: Unknown data has no data_source field. Use the "all" cleanup:

python3 scripts/cleanup_database.py --all --confirm

Issue: Cleanup Not Persisting

Solution: Check that data/captures_simple.json is writable:

ls -la data/captures_simple.json

Issue: Wrong Data Source Marked

Solution: Clean and re-import with correct marking:

python3 scripts/cleanup_database.py --source test
python3 scripts/import_wardriving_data.py data/ --data-source production

Summary

The data cleanup system allows you to:

• ✅ Mark uploads as test/mock/production
• ✅ Clean up by data source
• ✅ Clean up by session ID
• ✅ View statistics by data source
• ✅ Safely manage test data during development

Happy testing! 🧪