feat: Add wardriving data import system and cleanup tools
Major Features: - Multi-format GPS data import (Wigle CSV, GPS JSON, filename GPS) - Data source tracking (test/mock/production) - Database cleanup tools for managing test data - JSON file persistence for simplified server - UI improvements for map controls Backend: - Added wardriving_importer.py with WigleCSVImporter, GPSJSONImporter, BatchImporter - Added data_source and session_id tracking to captures - New API endpoints: DELETE /api/v1/admin/cleanup - Auto-save/load functionality for captures_simple.json - Updated stats endpoint to show data source breakdown CLI Tools: - scripts/import_wardriving_data.py - Batch import with --data-source flag - scripts/cleanup_database.py - Clean by source, session, or all - scripts/create_test_dataset.py - Generate GPS-tagged test data - scripts/test_wardriving_import.py - Test suite for importers Frontend: - Fixed map controls z-index and positioning issues - Moved controls to top-right to avoid zoom button overlap - Fixed Leaflet zoom controls rendering over header - Changed controls to position:fixed for persistent visibility - Export map object to window.map for proper invalidateSize Documentation: - docs/WARDRIVING_IMPORT.md - Complete import guide - docs/DATA_CLEANUP_GUIDE.md - Cleanup system documentation - docs/TEST_LOCATIONS.md - Test GPS coordinates reference - TEST_RESULTS_SUMMARY.md - Format testing results Test Data: - Created test_dataset_gps with 15 captures across 5 US cities - All test data marked with data_source="test" for easy cleanup Testing: - Verified Wigle CSV import (1 capture from West LA) - Verified GPS filename import (3 captures from UCLA area) - Verified GPS JSON companion files (15 captures, 5 cities) - Verified cleanup functionality (deleted 15 test captures) - Verified data persistence across server restarts 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,292 @@
|
||||
# Data Cleanup Guide
|
||||
|
||||
## Overview
|
||||
|
||||
GigLez now supports marking and cleaning up test/mock data from the database. This allows you to populate the platform with test data for development, then easily remove it when moving to production.
|
||||
|
||||
## Data Source Tracking
|
||||
|
||||
### Data Source Types
|
||||
|
||||
All captures are now tagged with a `data_source` field:
|
||||
|
||||
- **`test`** - Test data for development (default for CLI imports)
|
||||
- **`mock`** - Mock/synthetic data
|
||||
- **`production`** - Real wardriving data from field captures
|
||||
|
||||
### How Data Sources Are Assigned
|
||||
|
||||
#### During Upload (Web Interface)
|
||||
- Defaults to `production`
|
||||
- Can be overridden in upload manifest
|
||||
|
||||
#### During CLI Import
|
||||
```bash
|
||||
# Import as test data (default)
|
||||
python3 scripts/import_wardriving_data.py data/ --data-source test
|
||||
|
||||
# Import as mock data
|
||||
python3 scripts/import_wardriving_data.py data/ --data-source mock
|
||||
|
||||
# Import as production data
|
||||
python3 scripts/import_wardriving_data.py data/ --data-source production
|
||||
```
|
||||
|
||||
## Cleanup Methods
|
||||
|
||||
### Method 1: Cleanup Script (Recommended)
|
||||
|
||||
**Location**: `scripts/cleanup_database.py`
|
||||
|
||||
#### Clean by Data Source
|
||||
```bash
|
||||
# Delete all test data
|
||||
python3 scripts/cleanup_database.py --source test --stats
|
||||
|
||||
# Delete all mock data
|
||||
python3 scripts/cleanup_database.py --source mock --stats
|
||||
|
||||
# View stats before/after with --stats flag
|
||||
python3 scripts/cleanup_database.py --source test --stats
|
||||
```
|
||||
|
||||
#### Clean by Session ID
|
||||
```bash
|
||||
# Delete all captures from a specific import session
|
||||
python3 scripts/cleanup_database.py --session wardrive_import_15 --stats
|
||||
```
|
||||
|
||||
#### Clean All Data
|
||||
```bash
|
||||
# ⚠️ WARNING: Deletes EVERYTHING
|
||||
python3 scripts/cleanup_database.py --all --confirm --stats
|
||||
```
|
||||
|
||||
### Method 2: Direct API Calls
|
||||
|
||||
#### Delete by Data Source
|
||||
```bash
|
||||
# Delete test data
|
||||
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup?data_source=test"
|
||||
|
||||
# Response:
|
||||
{
|
||||
"success": true,
|
||||
"message": "Deleted 15 captures with data_source='test'",
|
||||
"deleted_count": 15,
|
||||
"remaining_count": 5
|
||||
}
|
||||
```
|
||||
|
||||
#### Delete by Session ID
|
||||
```bash
|
||||
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/session/wardrive_import_15"
|
||||
```
|
||||
|
||||
#### Delete All Data
|
||||
```bash
|
||||
curl -X DELETE "http://localhost:8000/api/v1/admin/cleanup/all"
|
||||
```
|
||||
|
||||
## Check Data Sources
|
||||
|
||||
### View Statistics
|
||||
```bash
|
||||
curl -s http://localhost:8000/api/v1/stats/summary | jq '{total_captures, data_sources}'
|
||||
```
|
||||
|
||||
**Example Output**:
|
||||
```json
|
||||
{
|
||||
"total_captures": 35,
|
||||
"data_sources": {
|
||||
"unknown": 20,
|
||||
"test": 15
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### List All Captures with Data Source
|
||||
```bash
|
||||
curl -s http://localhost:8000/api/v1/query/captures | jq '.captures[] | {filename, data_source}'
|
||||
```
|
||||
|
||||
## Typical Workflows
|
||||
|
||||
### Workflow 1: Development Testing
|
||||
|
||||
```bash
|
||||
# 1. Import test data
|
||||
python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/ --data-source test
|
||||
|
||||
# 2. Test features on web interface
|
||||
open http://localhost:8000
|
||||
|
||||
# 3. Clean up test data when done
|
||||
python3 scripts/cleanup_database.py --source test --stats
|
||||
```
|
||||
|
||||
### Workflow 2: Production Deployment
|
||||
|
||||
```bash
|
||||
# 1. Clean all test/mock data before production
|
||||
python3 scripts/cleanup_database.py --source test
|
||||
python3 scripts/cleanup_database.py --source mock
|
||||
|
||||
# 2. Import real wardriving data
|
||||
python3 scripts/import_wardriving_data.py real_captures/ --data-source production
|
||||
|
||||
# 3. Verify only production data remains
|
||||
curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
|
||||
```
|
||||
|
||||
### Workflow 3: Complete Reset
|
||||
|
||||
```bash
|
||||
# Delete EVERYTHING and start fresh
|
||||
python3 scripts/cleanup_database.py --all --confirm --stats
|
||||
```
|
||||
|
||||
## Migrating Existing Data
|
||||
|
||||
Existing captures without a `data_source` field will show as `"unknown"` in statistics.
|
||||
|
||||
To clean up unknown captures:
|
||||
```bash
|
||||
# Manually edit data/captures_simple.json and add "data_source" field
|
||||
# Or delete and re-import with proper marking
|
||||
python3 scripts/cleanup_database.py --all --confirm
|
||||
python3 scripts/import_wardriving_data.py data/ --data-source production
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### GET /api/v1/stats/summary
|
||||
Returns database statistics including data source breakdown:
|
||||
```json
|
||||
{
|
||||
"total_captures": 35,
|
||||
"unique_devices": 7,
|
||||
"data_sources": {
|
||||
"test": 15,
|
||||
"production": 20
|
||||
},
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### DELETE /api/v1/admin/cleanup?data_source={source}
|
||||
Delete captures by data source (`test`, `mock`, or `production`)
|
||||
|
||||
### DELETE /api/v1/admin/cleanup/session/{session_id}
|
||||
Delete all captures from a specific upload session
|
||||
|
||||
### DELETE /api/v1/admin/cleanup/all
|
||||
Delete ALL captures (use with extreme caution!)
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Import and Clean Test Data
|
||||
|
||||
```bash
|
||||
# Import 15 test captures
|
||||
$ python3 scripts/import_wardriving_data.py signatures/test_dataset_gps/
|
||||
📤 Uploading 15 captures to http://localhost:8000
|
||||
Data source: test
|
||||
...
|
||||
✅ Import complete!
|
||||
|
||||
# Check stats
|
||||
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
|
||||
{
|
||||
"test": 15
|
||||
}
|
||||
|
||||
# Clean up test data
|
||||
$ python3 scripts/cleanup_database.py --source test --stats
|
||||
BEFORE cleanup:
|
||||
📊 Current Database Stats:
|
||||
Total captures: 15
|
||||
By data source:
|
||||
test: 15
|
||||
|
||||
✅ Deleted 15 captures with data_source='test'
|
||||
Deleted: 15
|
||||
Remaining: 0
|
||||
|
||||
AFTER cleanup:
|
||||
📊 Current Database Stats:
|
||||
Total captures: 0
|
||||
```
|
||||
|
||||
### Example 2: Mixed Data Sources
|
||||
|
||||
```bash
|
||||
# Import test data
|
||||
$ python3 scripts/import_wardriving_data.py test_data/ --data-source test
|
||||
|
||||
# Import production data
|
||||
$ python3 scripts/import_wardriving_data.py real_data/ --data-source production
|
||||
|
||||
# Check distribution
|
||||
$ curl -s http://localhost:8000/api/v1/stats/summary | jq '.data_sources'
|
||||
{
|
||||
"test": 15,
|
||||
"production": 10
|
||||
}
|
||||
|
||||
# Delete only test data, keep production
|
||||
$ python3 scripts/cleanup_database.py --source test
|
||||
✅ Deleted 15 captures with data_source='test'
|
||||
Remaining: 10
|
||||
```
|
||||
|
||||
## Safety Features
|
||||
|
||||
1. **Confirmation Required**: `--all` cleanup requires `--confirm` flag
|
||||
2. **Data Source Validation**: Only accepts valid source types
|
||||
3. **Persistent Storage**: Changes are immediately saved to disk
|
||||
4. **Statistics Reporting**: Always shows deleted/remaining counts
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always use `--stats`** to see before/after counts
|
||||
2. **Mark test data clearly** using `--data-source test`
|
||||
3. **Use sessions** for grouping related imports
|
||||
4. **Backup production data** before mass deletions
|
||||
5. **Clean test data regularly** to avoid confusion
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Can't Delete Unknown Data
|
||||
|
||||
**Solution**: Unknown data has no `data_source` field. Use the "all" cleanup:
|
||||
```bash
|
||||
python3 scripts/cleanup_database.py --all --confirm
|
||||
```
|
||||
|
||||
### Issue: Cleanup Not Persisting
|
||||
|
||||
**Solution**: Check that `data/captures_simple.json` is writable:
|
||||
```bash
|
||||
ls -la data/captures_simple.json
|
||||
```
|
||||
|
||||
### Issue: Wrong Data Source Marked
|
||||
|
||||
**Solution**: Clean and re-import with correct marking:
|
||||
```bash
|
||||
python3 scripts/cleanup_database.py --source test
|
||||
python3 scripts/import_wardriving_data.py data/ --data-source production
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The data cleanup system allows you to:
|
||||
- ✅ Mark uploads as test/mock/production
|
||||
- ✅ Clean up by data source
|
||||
- ✅ Clean up by session ID
|
||||
- ✅ View statistics by data source
|
||||
- ✅ Safely manage test data during development
|
||||
|
||||
**Happy testing!** 🧪
|
||||
Reference in New Issue
Block a user