feat: RTL_433 protocol database import - iteration 1/5
- Expanded protocol database from 18 → 299 signatures (16.6x increase) - Imported 281 protocols from RTL_433 open-source database (286 total devices) - Created automated import script: scripts/import_rtl433_protocols.py - Generated rtl433_protocols_imported.py with timing/frequency/modulation data - Updated protocol_database.py to include RTL433_PROTOCOLS - All 26 tests passing Breakdown by category: - Weather: 116 protocols - Sensors: 36 protocols - TPMS: 25 protocols - Security: 23 protocols - Home Automation: 18 protocols - Other: 50+ protocols Frequency coverage: - 433.92 MHz: 248 protocols - 315.00 MHz: 32 protocols - 915.00 MHz: 1 protocol This provides comprehensive coverage of Sub-GHz IoT devices for accurate identification from raw RF captures.
This commit is contained in:
leetcrypt
@@ -0,0 +1,578 @@
|
||||
# Robust Error Logging System for Beta Testing
|
||||
|
||||
## Overview
|
||||
|
||||
Comprehensive error tracking and monitoring system for GigLez file uploads during beta testing. Captures detailed context at every stage of the upload process to help diagnose and fix issues quickly.
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### 1. **Server-Side Logging** (`src/logging/upload_logger.py`)
|
||||
|
||||
Centralized logging system that tracks:
|
||||
- Upload attempts with client info
|
||||
- Individual file processing stages
|
||||
- Parse errors with file samples
|
||||
- GPS validation failures
|
||||
- Device matching results
|
||||
- Manual labels
|
||||
- Performance metrics
|
||||
- Exceptions with full context
|
||||
|
||||
**Log Files Created:**
|
||||
```
|
||||
logs/uploads/
|
||||
├── uploads_2026-01-15.log # All events (DEBUG level)
|
||||
├── upload_errors_2026-01-15.log # Errors only
|
||||
├── uploads_structured_2026-01-15.jsonl # JSON for analysis
|
||||
├── upload_performance_2026-01-15.log # Performance metrics
|
||||
└── problem_files/ # Failed file samples
|
||||
└── upload_20260115_123456/
|
||||
└── problematic_file.sub.txt
|
||||
```
|
||||
|
||||
### 2. **Enhanced Upload Endpoint** (`src/api/routes/captures_with_logging.py`)
|
||||
|
||||
New beta endpoint: `POST /api/v1/captures/upload/beta`
|
||||
|
||||
Logs every step of the upload process:
|
||||
1. **Validation** - Manifest parsing, file count, total size
|
||||
2. **Parsing** - .sub file parsing with detailed error context
|
||||
3. **GPS Validation** - Coordinate validation
|
||||
4. **Storage** - File storage timing
|
||||
5. **Matching** - Device identification results
|
||||
6. **Database** - Record creation
|
||||
7. **Complete** - Final summary with performance metrics
|
||||
|
||||
### 3. **Client-Side Error Reporter** (`static/js/error_reporter.js`)
|
||||
|
||||
JavaScript error tracking that:
|
||||
- Captures uncaught exceptions
|
||||
- Reports parse errors with file samples
|
||||
- Sends network errors
|
||||
- Tracks browser/device info
|
||||
- Queues errors if offline
|
||||
|
||||
**Usage:**
|
||||
```javascript
|
||||
// Initialize (happens automatically)
|
||||
window.errorReporter = new ErrorReporter();
|
||||
|
||||
// Set upload ID for tracking
|
||||
errorReporter.setUploadId('upload_20260115_123456');
|
||||
|
||||
// Manually report errors
|
||||
try {
|
||||
parseSubFile(content);
|
||||
} catch (error) {
|
||||
errorReporter.reportParseError(filename, error, content);
|
||||
}
|
||||
```
|
||||
|
||||
### 4. **Log Analysis Tool** (`scripts/analyze_upload_logs.py`)
|
||||
|
||||
Command-line tool to analyze logs and generate reports.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### During Beta Testing
|
||||
|
||||
#### 1. **Enable Beta Endpoint**
|
||||
|
||||
Update upload form to use beta endpoint:
|
||||
|
||||
```javascript
|
||||
// In upload_enhanced.js
|
||||
const response = await fetch('/api/v1/captures/upload/beta', {
|
||||
method: 'POST',
|
||||
body: formData
|
||||
});
|
||||
```
|
||||
|
||||
#### 2. **Include Error Reporter**
|
||||
|
||||
Add to upload page HTML:
|
||||
|
||||
```html
|
||||
<!-- Error reporting -->
|
||||
<script src="/static/js/error_reporter.js"></script>
|
||||
|
||||
<!-- Before upload -->
|
||||
<script>
|
||||
// Generate upload ID
|
||||
const uploadId = errorReporter.generateUploadId();
|
||||
errorReporter.setUploadId(uploadId);
|
||||
|
||||
// Include in manifest
|
||||
manifest.upload_id = uploadId;
|
||||
</script>
|
||||
```
|
||||
|
||||
#### 3. **Monitor Logs in Real-Time**
|
||||
|
||||
```bash
|
||||
# Watch all uploads
|
||||
tail -f logs/uploads/uploads_$(date +%Y-%m-%d).log
|
||||
|
||||
# Watch errors only
|
||||
tail -f logs/uploads/upload_errors_$(date +%Y-%m-%d).log
|
||||
|
||||
# Watch structured JSON
|
||||
tail -f logs/uploads/uploads_structured_$(date +%Y-%m-%d).jsonl | jq
|
||||
```
|
||||
|
||||
### Analyzing Logs
|
||||
|
||||
#### Daily Summary
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_upload_logs.py --date 2026-01-15
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
================================================================================
|
||||
Upload Log Analysis - 2026-01-15
|
||||
================================================================================
|
||||
|
||||
📊 OVERVIEW
|
||||
Total Uploads: 152
|
||||
Total Files: 487
|
||||
Successful: 421 (91.2%)
|
||||
Failed: 41
|
||||
Duplicates: 25
|
||||
|
||||
👥 USERS
|
||||
Authenticated: 89
|
||||
Anonymous: 63
|
||||
|
||||
⚡ PERFORMANCE
|
||||
Parse Time: 45ms avg, 32ms median
|
||||
Matching Time: 128ms avg, 95ms median
|
||||
Total Time: 318ms avg, 256ms median
|
||||
|
||||
❌ TOP ERRORS
|
||||
SubFileParseError 23 occurrences
|
||||
GPSValidationError 12 occurrences
|
||||
NetworkError 6 occurrences
|
||||
|
||||
🔍 PARSE ERRORS (23)
|
||||
capture_bad.sub ValueError: Invalid frequency 0
|
||||
malformed.sub UnicodeDecodeError: invalid start byte
|
||||
```
|
||||
|
||||
#### Last 7 Days Trend
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_upload_logs.py --last 7
|
||||
```
|
||||
|
||||
#### Specific Upload Investigation
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_upload_logs.py --upload-id upload_20260115_123456
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
================================================================================
|
||||
Upload Details: upload_20260115_123456
|
||||
================================================================================
|
||||
|
||||
Total Events: 18
|
||||
|
||||
📅 TIMELINE:
|
||||
10:23:15.234 | upload_attempt | 5 files, 2.3 MB
|
||||
10:23:15.456 | file_processing | ✓ capture_001.sub - parsing (42ms)
|
||||
10:23:15.512 | file_processing | ✓ capture_001.sub - matching (95ms)
|
||||
10:23:15.623 | matching_results | 🔍 3 matches (95ms)
|
||||
10:23:15.701 | file_processing | ✓ capture_001.sub - complete (267ms)
|
||||
10:23:15.889 | parse_error | ❌ capture_002.sub - ValueError
|
||||
10:23:16.234 | upload_complete | ✅ 4 success, 1 failed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Log Entry Types
|
||||
|
||||
### 1. **Upload Attempt**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "upload_attempt",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:15.234Z",
|
||||
"user_id": 42,
|
||||
"session_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"file_count": 5,
|
||||
"total_size_bytes": 2415360,
|
||||
"total_size_mb": 2.3,
|
||||
"client_info": {
|
||||
"user_agent": "Mozilla/5.0 ...",
|
||||
"ip_address": "192.168.1.100",
|
||||
"referer": "https://giglez.com/upload"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. **File Processing**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "file_processing",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:15.456Z",
|
||||
"filename": "capture_001.sub",
|
||||
"file_size": 512,
|
||||
"file_hash": "abc123...",
|
||||
"stage": "parsing",
|
||||
"success": true,
|
||||
"duration_ms": 42.15,
|
||||
"metadata": {
|
||||
"frequency": 433920000,
|
||||
"protocol": "Princeton",
|
||||
"modulation": "OOK"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. **Parse Error**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "parse_error",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:15.889Z",
|
||||
"filename": "capture_002.sub",
|
||||
"file_hash": "def456...",
|
||||
"error_type": "ValueError",
|
||||
"error_message": "Invalid frequency: 0",
|
||||
"file_content_sample": "Filetype: Flipper SubGhz Key File\\nVersion: 1\\nFrequency: 0\\n...",
|
||||
"traceback": "Traceback (most recent call last):\\n File ..."
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** Full file sample saved to `logs/uploads/problem_files/upload_20260115_123456/capture_002.sub.txt`
|
||||
|
||||
### 4. **GPS Validation Error**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "gps_validation_error",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:16.001Z",
|
||||
"filename": "capture_003.sub",
|
||||
"latitude": 999.0,
|
||||
"longitude": -200.5,
|
||||
"accuracy": 5.0,
|
||||
"reason": "Coordinates out of range or invalid"
|
||||
}
|
||||
```
|
||||
|
||||
### 5. **Matching Results**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "matching_results",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:15.512Z",
|
||||
"filename": "capture_001.sub",
|
||||
"file_hash": "abc123...",
|
||||
"match_count": 3,
|
||||
"top_match": {
|
||||
"device_id": 123,
|
||||
"manufacturer": "Chamberlain",
|
||||
"model": "891LM",
|
||||
"confidence": 0.95
|
||||
},
|
||||
"all_matches": [...],
|
||||
"duration_ms": 95.23,
|
||||
"matcher_stats": {
|
||||
"strategies_used": 6,
|
||||
"total_candidates": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6. **Performance Metrics**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "performance_metrics",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:16.234Z",
|
||||
"avg_parse_time_ms": 45.2,
|
||||
"avg_matching_time_ms": 128.7,
|
||||
"avg_storage_time_ms": 12.3,
|
||||
"avg_db_time_ms": 23.1,
|
||||
"total_upload_time_ms": 1234.5
|
||||
}
|
||||
```
|
||||
|
||||
### 7. **Client Error**
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "client_error",
|
||||
"upload_id": "upload_20260115_123456",
|
||||
"timestamp": "2026-01-15T10:23:16.500Z",
|
||||
"error_type": "SubFileParseError",
|
||||
"error_message": "Cannot read property 'frequency' of undefined",
|
||||
"client_info": {
|
||||
"user_agent": "Mozilla/5.0 ...",
|
||||
"platform": "Linux x86_64",
|
||||
"screen_width": 1920,
|
||||
"screen_height": 1080,
|
||||
"connection_type": "4g",
|
||||
"device_memory": 8
|
||||
},
|
||||
"stack_trace": "at parseSubFile (sub_parser.js:42:15)\\n..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
### Issue: High Parse Error Rate
|
||||
|
||||
**Symptoms:** 20%+ of files failing to parse
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
python3 scripts/analyze_upload_logs.py --date 2026-01-15 | grep "PARSE ERRORS" -A 10
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. Check `logs/uploads/problem_files/` for actual file samples
|
||||
2. Identify common error patterns
|
||||
3. Update parser to handle edge cases
|
||||
4. Add validation warnings in upload UI
|
||||
|
||||
### Issue: Slow Matching Performance
|
||||
|
||||
**Symptoms:** Matching taking >500ms average
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
python3 scripts/analyze_upload_logs.py --date 2026-01-15 | grep "Matching Time"
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. Check database indexes on `signatures` table
|
||||
2. Review matcher strategy order (put fast strategies first)
|
||||
3. Consider caching frequent patterns
|
||||
4. Profile matching code with `cProfile`
|
||||
|
||||
### Issue: GPS Validation Failures
|
||||
|
||||
**Symptoms:** Users reporting valid coordinates being rejected
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
grep "gps_validation_error" logs/uploads/uploads_structured_2026-01-15.jsonl | jq
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. Review coordinate ranges in validator
|
||||
2. Check for precision issues (too many decimal places)
|
||||
3. Add client-side validation to catch before upload
|
||||
4. Provide clear error messages
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Dashboard (Future)
|
||||
|
||||
Recommended metrics to track:
|
||||
|
||||
1. **Upload Success Rate** (daily)
|
||||
- Target: >95%
|
||||
- Alert if <90%
|
||||
|
||||
2. **Average Processing Time** (hourly)
|
||||
- Target: <500ms
|
||||
- Alert if >1000ms
|
||||
|
||||
3. **Error Rate by Type** (hourly)
|
||||
- Track top 5 error types
|
||||
- Alert on new error types
|
||||
|
||||
4. **User Impact** (daily)
|
||||
- % of users experiencing errors
|
||||
- % of files failing per user
|
||||
|
||||
5. **Geographic Distribution** (weekly)
|
||||
- Upload sources by country
|
||||
- Network quality by region
|
||||
|
||||
---
|
||||
|
||||
## Retention Policy
|
||||
|
||||
| Log Type | Retention | Reason |
|
||||
|----------|-----------|--------|
|
||||
| Main uploads log | 90 days | Debugging window |
|
||||
| Error log | 180 days | Long-term analysis |
|
||||
| Structured JSON | 90 days | Analysis queries |
|
||||
| Performance log | 30 days | Recent trends only |
|
||||
| Problem file samples | 30 days | Storage constraints |
|
||||
|
||||
**Cleanup:**
|
||||
```bash
|
||||
# Remove logs older than 90 days
|
||||
find logs/uploads -name "uploads_*.log" -mtime +90 -delete
|
||||
|
||||
# Archive before deletion
|
||||
tar -czf logs/archive/uploads_2026-01.tar.gz logs/uploads/uploads_2026-01-*.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Privacy Considerations
|
||||
|
||||
### What We Log:
|
||||
- File metadata (frequency, protocol, size)
|
||||
- GPS coordinates (as submitted)
|
||||
- User ID (if authenticated)
|
||||
- IP address
|
||||
- Browser/device info
|
||||
|
||||
### What We Don't Log:
|
||||
- Full file contents (only first 500 chars on error)
|
||||
- Passwords or tokens
|
||||
- Personal identifying information beyond user ID
|
||||
|
||||
### GDPR Compliance:
|
||||
- Users can request deletion of their upload logs
|
||||
- IP addresses can be anonymized (last octet masked)
|
||||
- Option to disable client-side error reporting
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### For Developers:
|
||||
|
||||
1. **Always use `upload_id`** to correlate logs
|
||||
2. **Log before and after** critical operations
|
||||
3. **Include context** in exception logs (filename, user_id, etc.)
|
||||
4. **Use appropriate log levels**:
|
||||
- DEBUG: Detailed diagnostic info
|
||||
- INFO: General events
|
||||
- WARNING: Potential issues
|
||||
- ERROR: Failures requiring attention
|
||||
- CRITICAL: System-wide failures
|
||||
|
||||
### For Beta Testers:
|
||||
|
||||
1. **Report upload IDs** when filing bug reports
|
||||
2. **Include browser/device info** (captured automatically)
|
||||
3. **Save screenshots** of error messages
|
||||
4. **Note reproduction steps**
|
||||
|
||||
### For Operations:
|
||||
|
||||
1. **Monitor error logs daily** during beta
|
||||
2. **Set up alerts** for error rate spikes
|
||||
3. **Review problem file samples** weekly
|
||||
4. **Archive old logs** monthly
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Server-Side Logging Functions
|
||||
|
||||
```python
|
||||
from src.logging.upload_logger import (
|
||||
log_upload_attempt,
|
||||
log_file_processing,
|
||||
log_parse_error,
|
||||
log_gps_validation_error,
|
||||
log_matching_results,
|
||||
log_manual_label,
|
||||
log_upload_complete,
|
||||
log_exception,
|
||||
log_performance_metrics,
|
||||
log_client_error,
|
||||
UploadStage
|
||||
)
|
||||
|
||||
# Example: Log parse error
|
||||
log_parse_error(
|
||||
upload_id="upload_20260115_123456",
|
||||
filename="capture.sub",
|
||||
file_content_sample=content[:500],
|
||||
error_type="ValueError",
|
||||
error_message=str(e),
|
||||
traceback_str=traceback.format_exc(),
|
||||
file_hash="abc123..."
|
||||
)
|
||||
```
|
||||
|
||||
### Client-Side Error Reporter
|
||||
|
||||
```javascript
|
||||
// Global instance
|
||||
window.errorReporter
|
||||
|
||||
// Set upload ID
|
||||
errorReporter.setUploadId('upload_20260115_123456');
|
||||
|
||||
// Report errors
|
||||
errorReporter.reportParseError(filename, error, fileContentSample);
|
||||
errorReporter.reportUploadError(filename, error, stage);
|
||||
errorReporter.reportValidationError(field, value, reason);
|
||||
errorReporter.reportNetworkError(url, status, statusText);
|
||||
|
||||
// Flush error queue
|
||||
await errorReporter.flushErrorQueue();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Logs not appearing?
|
||||
|
||||
**Check:**
|
||||
1. Log directory exists: `mkdir -p logs/uploads`
|
||||
2. Write permissions: `chmod 755 logs/uploads`
|
||||
3. Loguru installed: `pip install loguru`
|
||||
4. Logger initialized: Check `src/logging/upload_logger.py` import
|
||||
|
||||
### Analysis script failing?
|
||||
|
||||
**Common issues:**
|
||||
- Missing log files for date
|
||||
- JSON parse errors in structured log
|
||||
- Python version <3.7 (need 3.7+ for dataclasses)
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
# Check log files exist
|
||||
ls -la logs/uploads/
|
||||
|
||||
# Validate JSON
|
||||
cat logs/uploads/uploads_structured_2026-01-15.jsonl | jq . > /dev/null
|
||||
|
||||
# Install dependencies
|
||||
pip install loguru
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
**Questions or Issues:**
|
||||
- GitHub: `github.com/your-org/giglez/issues`
|
||||
- Email: support@giglez.com
|
||||
- Slack: #giglez-beta-testing
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-16
|
||||
**Version:** 1.0
|
||||
Reference in New Issue
Block a user