feat: RTL_433 protocol database import - iteration 1/5
- Expanded protocol database from 18 → 299 signatures (16.6x increase) - Imported 281 protocols from RTL_433 open-source database (286 total devices) - Created automated import script: scripts/import_rtl433_protocols.py - Generated rtl433_protocols_imported.py with timing/frequency/modulation data - Updated protocol_database.py to include RTL433_PROTOCOLS - All 26 tests passing Breakdown by category: - Weather: 116 protocols - Sensors: 36 protocols - TPMS: 25 protocols - Security: 23 protocols - Home Automation: 18 protocols - Other: 50+ protocols Frequency coverage: - 433.92 MHz: 248 protocols - 315.00 MHz: 32 protocols - 915.00 MHz: 1 protocol This provides comprehensive coverage of Sub-GHz IoT devices for accurate identification from raw RF captures.
This commit is contained in:
leetcrypt
@@ -0,0 +1,808 @@
|
||||
# ML/Neural Network Design for RF Signal Classification
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document outlines the machine learning approach for automatic RF device identification in GigLez, focusing on Sub-GHz IoT signals (300-928 MHz). The design prioritizes practical deployment using JavaScript/TensorFlow.js for browser-based inference, with Python training pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Research Foundation
|
||||
|
||||
### Key Papers & Approaches
|
||||
|
||||
1. **Deep Learning for RF Signal Classification** (Shi & Davaslioglu, 2019)
|
||||
- CNN architectures for modulation recognition
|
||||
- Achieves >95% accuracy above 2dB SNR
|
||||
- Uses IQ sample input (128-256 samples)
|
||||
|
||||
2. **RF Fingerprinting for IoT** (Jian et al., 2020)
|
||||
- Device-specific transmitter signatures
|
||||
- Massive experimental study on real IoT devices
|
||||
- Addresses resource-constrained edge deployment
|
||||
|
||||
3. **Practical RF Machine Learning** (Panoradio SDR)
|
||||
- CNN + RNN hybrid architectures
|
||||
- Categorical cross-entropy optimization
|
||||
- Real-world performance validation
|
||||
|
||||
### State-of-the-Art Insights
|
||||
|
||||
**What Works:**
|
||||
- CNNs excel at extracting spatial features from signal representations
|
||||
- 1D CNNs on raw pulse data perform well for timing-based protocols
|
||||
- 2D CNNs on spectrograms capture frequency-domain patterns
|
||||
- Ensemble methods (CNN + heuristics) outperform pure ML
|
||||
|
||||
**Challenges:**
|
||||
- Single-transmission captures (Flipper Zero) vs. continuous streams (RTL-SDR)
|
||||
- Variable signal lengths require padding or dynamic architectures
|
||||
- Class imbalance (1000+ garage openers, <10 weather sensors)
|
||||
- Synthetic vs. real-world signal distribution shift
|
||||
|
||||
---
|
||||
|
||||
## GigLez-Specific Requirements
|
||||
|
||||
### Input Characteristics
|
||||
|
||||
**Signal Types:**
|
||||
1. **Decoded (.sub KEY format)**
|
||||
- Protocol, frequency, bit length, key data, timing element
|
||||
- Structured metadata, high information density
|
||||
- Best for heuristic matching (current system)
|
||||
|
||||
2. **RAW (.sub RAW format)**
|
||||
- Pulse/gap timing arrays (e.g., `2980 -240 520 -980 ...`)
|
||||
- Variable length (10-10,000+ samples)
|
||||
- Target for ML approach
|
||||
|
||||
3. **BinRAW (.sub BinRAW format)**
|
||||
- Binary pulse data
|
||||
- Less common, lower priority
|
||||
|
||||
**Challenge:** 97% of Flipper Zero captures are single-transmission (1-3 repeats), not continuous streams.
|
||||
|
||||
### Output Requirements
|
||||
|
||||
**Device Classification:**
|
||||
- Primary: Device type (e.g., "Garage Door Opener", "Weather Sensor - Temperature")
|
||||
- Secondary: Manufacturer (e.g., "Chamberlain", "Oregon Scientific")
|
||||
- Tertiary: Model (e.g., "LiftMaster 891LM", "THGN123N")
|
||||
|
||||
**Confidence Scoring:**
|
||||
- Probabilistic output (softmax)
|
||||
- Combine with heuristic matcher scores
|
||||
- Threshold: >0.7 for auto-ID, 0.5-0.7 for suggestions, <0.5 unknown
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture: Hybrid Ensemble System
|
||||
|
||||
### Overview
|
||||
|
||||
Combine **three parallel classifiers** with weighted ensemble:
|
||||
|
||||
```
|
||||
RAW .sub file
|
||||
↓
|
||||
[Feature Extraction]
|
||||
├─→ 1D CNN (Temporal Features) [35% weight]
|
||||
├─→ Statistical Feature Classifier [25% weight]
|
||||
└─→ Heuristic Matcher (existing) [40% weight]
|
||||
↓
|
||||
[Weighted Ensemble]
|
||||
↓
|
||||
Device ID + Confidence
|
||||
```
|
||||
|
||||
**Rationale:**
|
||||
- Heuristic matcher handles decoded signals perfectly (1.0 confidence)
|
||||
- CNN learns patterns in RAW signals the heuristics miss
|
||||
- Statistical features provide interpretable fallback
|
||||
- Weighted ensemble prevents over-reliance on any single method
|
||||
|
||||
---
|
||||
|
||||
## Model 1: 1D CNN for Pulse Timing Classification
|
||||
|
||||
### Architecture
|
||||
|
||||
**Input:**
|
||||
- RAW pulse data, fixed length 512 samples (padded/truncated)
|
||||
- Normalization: Divide by max(abs(value)) to range [-1, 1]
|
||||
- Shape: `(batch_size, 512, 1)`
|
||||
|
||||
**Network Structure:**
|
||||
|
||||
```python
|
||||
# TensorFlow.js equivalent structure
|
||||
model = Sequential([
|
||||
# Block 1: Initial feature extraction
|
||||
Conv1D(64, kernel_size=16, strides=2, activation='relu', padding='same'),
|
||||
BatchNormalization(),
|
||||
MaxPooling1D(pool_size=4),
|
||||
Dropout(0.2),
|
||||
|
||||
# Block 2: Mid-level features
|
||||
Conv1D(128, kernel_size=8, strides=2, activation='relu', padding='same'),
|
||||
BatchNormalization(),
|
||||
MaxPooling1D(pool_size=4),
|
||||
Dropout(0.3),
|
||||
|
||||
# Block 3: High-level features
|
||||
Conv1D(256, kernel_size=4, strides=2, activation='relu', padding='same'),
|
||||
BatchNormalization(),
|
||||
GlobalAveragePooling1D(),
|
||||
|
||||
# Classification head
|
||||
Dense(512, activation='relu'),
|
||||
Dropout(0.4),
|
||||
Dense(num_classes, activation='softmax')
|
||||
])
|
||||
```
|
||||
|
||||
**Hyperparameters:**
|
||||
- Optimizer: Adam (lr=0.001, decay=1e-6)
|
||||
- Loss: Categorical cross-entropy with label smoothing (0.1)
|
||||
- Batch size: 32
|
||||
- Epochs: 50 with early stopping (patience=5)
|
||||
|
||||
**Data Augmentation:**
|
||||
- Random time stretching (0.9x - 1.1x)
|
||||
- Gaussian noise injection (SNR 20-40 dB)
|
||||
- Random clipping (simulate distance variations)
|
||||
|
||||
**Output:**
|
||||
- Softmax probabilities for top 100 device classes
|
||||
- Classes determined by dataset frequency (>50 examples)
|
||||
|
||||
---
|
||||
|
||||
## Model 2: Statistical Feature Classifier
|
||||
|
||||
### Feature Extraction (Handcrafted)
|
||||
|
||||
From RAW pulse data, extract 47 features:
|
||||
|
||||
**Timing Features (16):**
|
||||
- Mean, median, std, min, max of positive pulses
|
||||
- Mean, median, std, min, max of negative gaps
|
||||
- Pulse/gap ratio, duty cycle
|
||||
- Short pulse width, long pulse width (K-means clusters)
|
||||
- Coefficient of variation for pulses and gaps
|
||||
|
||||
**Frequency Domain (12):**
|
||||
- FFT magnitude peaks (top 5 frequencies)
|
||||
- Dominant frequency, total energy, spectral centroid
|
||||
- Bandwidth (95% energy containment)
|
||||
|
||||
**Pattern Features (10):**
|
||||
- Autocorrelation peaks (indicates repeating patterns)
|
||||
- Zero-crossing rate
|
||||
- Peak count, valley count
|
||||
- Longest run of similar-width pulses
|
||||
- Entropy of pulse width distribution
|
||||
|
||||
**Metadata Features (9):**
|
||||
- Frequency (normalized to 433/868/315/915 bands)
|
||||
- Modulation type (one-hot: OOK, FSK, ASK)
|
||||
- Total pulse count
|
||||
- Total duration (ms)
|
||||
- Average bit rate (estimated)
|
||||
|
||||
### Classifier
|
||||
|
||||
**Algorithm:** Gradient Boosted Trees (LightGBM or XGBoost)
|
||||
|
||||
**Rationale:**
|
||||
- Handles tabular features excellently
|
||||
- Fast inference (critical for browser deployment)
|
||||
- Interpretable feature importance
|
||||
- Robust to missing values
|
||||
|
||||
**Training:**
|
||||
```python
|
||||
import lightgbm as lgb
|
||||
|
||||
params = {
|
||||
'objective': 'multiclass',
|
||||
'num_class': num_classes,
|
||||
'metric': 'multi_logloss',
|
||||
'boosting_type': 'gbdt',
|
||||
'num_leaves': 63,
|
||||
'learning_rate': 0.05,
|
||||
'feature_fraction': 0.8,
|
||||
'bagging_fraction': 0.8,
|
||||
'bagging_freq': 5,
|
||||
'max_depth': 8
|
||||
}
|
||||
|
||||
model = lgb.train(params, train_data, num_boost_round=500)
|
||||
```
|
||||
|
||||
**Export:** Convert to ONNX for browser inference via onnxruntime-web
|
||||
|
||||
---
|
||||
|
||||
## Model 3: Heuristic Matcher (Existing System)
|
||||
|
||||
**Keep current implementation** from `src/matcher/`:
|
||||
- ExactMatcher, PartialMatcher, PatternMatcher, etc.
|
||||
- Confidence scores: 0.5-1.0
|
||||
- Fast, deterministic, handles decoded signals
|
||||
|
||||
**Integration:** Run in parallel with ML models, ensemble at end
|
||||
|
||||
---
|
||||
|
||||
## Ensemble Strategy
|
||||
|
||||
### Weighted Voting
|
||||
|
||||
```javascript
|
||||
function ensemblePredict(cnn_probs, stat_probs, heuristic_results) {
|
||||
const weights = {
|
||||
cnn: 0.35,
|
||||
statistical: 0.25,
|
||||
heuristic: 0.40
|
||||
};
|
||||
|
||||
// Combine probabilities
|
||||
let combined = {};
|
||||
|
||||
// CNN contribution
|
||||
for (let [device_id, prob] of Object.entries(cnn_probs)) {
|
||||
combined[device_id] = (combined[device_id] || 0) + prob * weights.cnn;
|
||||
}
|
||||
|
||||
// Statistical contribution
|
||||
for (let [device_id, prob] of Object.entries(stat_probs)) {
|
||||
combined[device_id] = (combined[device_id] || 0) + prob * weights.statistical;
|
||||
}
|
||||
|
||||
// Heuristic contribution (convert confidence to probability)
|
||||
for (let match of heuristic_results) {
|
||||
let device_id = match.device_id;
|
||||
let prob = match.confidence; // Already 0-1 range
|
||||
combined[device_id] = (combined[device_id] || 0) + prob * weights.heuristic;
|
||||
}
|
||||
|
||||
// Sort by combined score
|
||||
let sorted = Object.entries(combined)
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 10); // Top 10
|
||||
|
||||
return sorted.map(([device_id, score]) => ({
|
||||
device_id: device_id,
|
||||
confidence: score,
|
||||
method: 'ensemble'
|
||||
}));
|
||||
}
|
||||
```
|
||||
|
||||
### Dynamic Weighting
|
||||
|
||||
Adjust weights based on input signal type:
|
||||
|
||||
| Signal Type | CNN Weight | Statistical Weight | Heuristic Weight |
|
||||
|-------------|------------|--------------------| -----------------|
|
||||
| Decoded (KEY format) | 0.10 | 0.10 | 0.80 |
|
||||
| RAW with >500 samples | 0.45 | 0.25 | 0.30 |
|
||||
| RAW with <100 samples | 0.20 | 0.40 | 0.40 |
|
||||
| Known protocol detected | 0.05 | 0.05 | 0.90 |
|
||||
|
||||
---
|
||||
|
||||
## Training Data Pipeline
|
||||
|
||||
### Dataset Structure
|
||||
|
||||
**Source:** Downloaded datasets (41,334 .sub files)
|
||||
|
||||
**Labeling Strategy:**
|
||||
|
||||
1. **Automatic Labels (High Confidence)**
|
||||
- Use file path structure (e.g., `Weather_stations/Oregon_Scientific/...`)
|
||||
- Decoded protocol name match
|
||||
- RTL_433 successful decode
|
||||
- **Estimated:** 15,000 high-quality labels
|
||||
|
||||
2. **Manual Labels (User-Submitted)**
|
||||
- Upload form includes device selection
|
||||
- Community verification voting
|
||||
- Store in `manual_identifications` table
|
||||
- **Expected:** 1,000+ labels over 6 months
|
||||
|
||||
3. **Semi-Supervised Learning**
|
||||
- Use high-confidence heuristic matches (1.0 confidence) as pseudo-labels
|
||||
- Iterative refinement: Train model → Generate predictions → Review low-confidence → Retrain
|
||||
- **Potential:** 25,000+ samples
|
||||
|
||||
### Class Distribution Handling
|
||||
|
||||
**Problem:** Long-tail distribution
|
||||
- Top 10 classes: 60% of data (garage door openers)
|
||||
- Bottom 50 classes: 5% of data (weather sensors, TPMS)
|
||||
|
||||
**Solutions:**
|
||||
1. **Class Weighting:**
|
||||
```python
|
||||
from sklearn.utils.class_weight import compute_class_weight
|
||||
|
||||
class_weights = compute_class_weight(
|
||||
'balanced',
|
||||
classes=np.unique(y_train),
|
||||
y=y_train
|
||||
)
|
||||
```
|
||||
|
||||
2. **Focal Loss:** Penalize easy examples (common classes)
|
||||
```python
|
||||
def focal_loss(y_true, y_pred, alpha=0.25, gamma=2.0):
|
||||
ce = -y_true * np.log(y_pred + 1e-7)
|
||||
weight = alpha * y_true * np.power(1 - y_pred, gamma)
|
||||
return np.sum(weight * ce)
|
||||
```
|
||||
|
||||
3. **Stratified Sampling:** Ensure rare classes in each batch
|
||||
|
||||
### Train/Val/Test Split
|
||||
|
||||
```python
|
||||
from sklearn.model_selection import StratifiedShuffleSplit
|
||||
|
||||
splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
|
||||
train_idx, test_idx = next(splitter.split(X, y))
|
||||
|
||||
# Further split training into train/val
|
||||
splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.15, random_state=42)
|
||||
train_idx, val_idx = next(splitter.split(X[train_idx], y[train_idx]))
|
||||
```
|
||||
|
||||
**Split:**
|
||||
- Training: 68% (~28,000 samples)
|
||||
- Validation: 12% (~5,000 samples)
|
||||
- Test: 20% (~8,000 samples)
|
||||
|
||||
---
|
||||
|
||||
## JavaScript Deployment (TensorFlow.js)
|
||||
|
||||
### Browser-Based Inference
|
||||
|
||||
**Why Browser:**
|
||||
- Instant feedback during upload
|
||||
- No server cost for inference
|
||||
- Privacy: .sub files never leave device for pre-screening
|
||||
- Offline capability (PWA)
|
||||
|
||||
### Model Conversion
|
||||
|
||||
**1. Train in Python (TensorFlow/Keras)**
|
||||
```python
|
||||
model.save('models/cnn_classifier_v1.h5')
|
||||
```
|
||||
|
||||
**2. Convert to TensorFlow.js**
|
||||
```bash
|
||||
tensorflowjs_converter \
|
||||
--input_format=keras \
|
||||
--output_format=tfjs_graph_model \
|
||||
models/cnn_classifier_v1.h5 \
|
||||
static/models/cnn_v1/
|
||||
```
|
||||
|
||||
**3. Load in Browser**
|
||||
```javascript
|
||||
import * as tf from '@tensorflow/tfjs';
|
||||
|
||||
const model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');
|
||||
|
||||
// Inference
|
||||
function predict(rawPulseArray) {
|
||||
// Preprocess
|
||||
let normalized = rawPulseArray.map(x => x / Math.max(...rawPulseArray.map(Math.abs)));
|
||||
let padded = padOrTruncate(normalized, 512);
|
||||
|
||||
// Convert to tensor
|
||||
let tensor = tf.tensor3d([padded.map(x => [x])], [1, 512, 1]);
|
||||
|
||||
// Predict
|
||||
let probs = model.predict(tensor);
|
||||
let probsArray = await probs.data();
|
||||
|
||||
// Get top 5
|
||||
let indexed = Array.from(probsArray).map((p, i) => [i, p]);
|
||||
indexed.sort((a, b) => b[1] - a[1]);
|
||||
|
||||
return indexed.slice(0, 5).map(([idx, prob]) => ({
|
||||
device_id: classMapping[idx],
|
||||
confidence: prob
|
||||
}));
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
**Model Quantization:**
|
||||
```bash
|
||||
tensorflowjs_converter \
|
||||
--input_format=keras \
|
||||
--output_format=tfjs_graph_model \
|
||||
--quantization_bytes=2 \
|
||||
models/cnn_classifier_v1.h5 \
|
||||
static/models/cnn_v1_quantized/
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- 4x smaller model (16-bit weights vs 32-bit)
|
||||
- 2x faster inference on mobile
|
||||
- Minimal accuracy loss (<1%)
|
||||
|
||||
**Web Workers:**
|
||||
```javascript
|
||||
// Main thread
|
||||
const worker = new Worker('/static/js/ml_worker.js');
|
||||
|
||||
worker.postMessage({
|
||||
type: 'predict',
|
||||
raw_data: pulseArray
|
||||
});
|
||||
|
||||
worker.onmessage = (e) => {
|
||||
console.log('Predictions:', e.data.predictions);
|
||||
};
|
||||
|
||||
// Worker thread (ml_worker.js)
|
||||
importScripts('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs');
|
||||
|
||||
let model = null;
|
||||
|
||||
self.onmessage = async (e) => {
|
||||
if (!model) {
|
||||
model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');
|
||||
}
|
||||
|
||||
const predictions = predict(e.data.raw_data);
|
||||
self.postMessage({ predictions });
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Statistical Feature Classifier Deployment
|
||||
|
||||
### ONNX Runtime Web
|
||||
|
||||
**1. Train in Python (LightGBM)**
|
||||
```python
|
||||
import lightgbm as lgb
|
||||
import onnxmltools
|
||||
from onnxmltools.convert import convert_lightgbm
|
||||
|
||||
# Train model
|
||||
model = lgb.train(params, train_data)
|
||||
|
||||
# Convert to ONNX
|
||||
onnx_model = convert_lightgbm(
|
||||
model,
|
||||
initial_types=[('input', FloatTensorType([None, 47]))],
|
||||
target_opset=12
|
||||
)
|
||||
|
||||
onnxmltools.utils.save_model(onnx_model, 'models/stat_classifier_v1.onnx')
|
||||
```
|
||||
|
||||
**2. Load in Browser**
|
||||
```javascript
|
||||
import * as ort from 'onnxruntime-web';
|
||||
|
||||
const session = await ort.InferenceSession.create('/static/models/stat_v1.onnx');
|
||||
|
||||
async function predictStatistical(features) {
|
||||
// features: Float32Array of length 47
|
||||
const tensor = new ort.Tensor('float32', features, [1, 47]);
|
||||
const feeds = { input: tensor };
|
||||
const results = await session.run(feeds);
|
||||
|
||||
// results.probabilities: [1, num_classes]
|
||||
const probs = results.probabilities.data;
|
||||
|
||||
return Array.from(probs).map((prob, idx) => ({
|
||||
device_id: classMapping[idx],
|
||||
confidence: prob
|
||||
})).sort((a, b) => b.confidence - a.confidence).slice(0, 5);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Training Schedule & Versioning
|
||||
|
||||
### Initial Training (Week 1)
|
||||
|
||||
**Goal:** Establish baseline
|
||||
|
||||
1. Label 5,000 high-confidence samples (automatic + file paths)
|
||||
2. Train CNN on 50 most common classes
|
||||
3. Train statistical classifier on same 50 classes
|
||||
4. Export to TensorFlow.js and ONNX
|
||||
5. Deploy v1.0 models
|
||||
|
||||
**Metrics:**
|
||||
- Top-1 accuracy: Target >70%
|
||||
- Top-5 accuracy: Target >85%
|
||||
- Inference time: <200ms on desktop, <500ms mobile
|
||||
|
||||
### Continuous Improvement (Ongoing)
|
||||
|
||||
**Weekly Retraining:**
|
||||
1. Collect new manual labels from upload form
|
||||
2. Review confidence scores on production data
|
||||
3. Add low-confidence samples to manual review queue
|
||||
4. Retrain models with expanded dataset
|
||||
5. A/B test new model vs. current model
|
||||
6. Deploy if metrics improve by >2%
|
||||
|
||||
**Version Tracking:**
|
||||
```
|
||||
models/
|
||||
├── cnn/
|
||||
│ ├── v1.0_2026-01-15/
|
||||
│ │ ├── model.json
|
||||
│ │ ├── group1-shard1of2.bin
|
||||
│ │ ├── group1-shard2of2.bin
|
||||
│ │ ├── metadata.json (accuracy, classes, date)
|
||||
│ │ └── training_log.txt
|
||||
│ ├── v1.1_2026-01-22/
|
||||
│ └── latest → v1.1_2026-01-22
|
||||
├── statistical/
|
||||
│ ├── v1.0_2026-01-15/
|
||||
│ │ ├── model.onnx
|
||||
│ │ ├── feature_metadata.json
|
||||
│ │ └── training_log.txt
|
||||
│ └── latest → v1.0_2026-01-15
|
||||
└── ensemble_config.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Metrics
|
||||
|
||||
### Model Performance
|
||||
|
||||
**Classification Metrics:**
|
||||
- Top-1 Accuracy
|
||||
- Top-5 Accuracy
|
||||
- Macro-averaged F1 (handle class imbalance)
|
||||
- Per-class Precision/Recall (focus on rare classes)
|
||||
|
||||
**Confidence Calibration:**
|
||||
- Expected Calibration Error (ECE)
|
||||
- Reliability diagram (predicted prob vs. actual accuracy)
|
||||
|
||||
**Ensemble Performance:**
|
||||
- Improvement over best single model
|
||||
- Disagreement analysis (where models differ)
|
||||
|
||||
### Production Metrics
|
||||
|
||||
**User Behavior:**
|
||||
- Auto-accept rate (user confirms ML prediction without editing)
|
||||
- Manual correction rate
|
||||
- "Unknown device" submission rate
|
||||
|
||||
**System Performance:**
|
||||
- Inference latency (p50, p95, p99)
|
||||
- Model download size
|
||||
- Cache hit rate
|
||||
|
||||
**Data Quality:**
|
||||
- Label confidence distribution
|
||||
- Inter-annotator agreement (when multiple users label same capture)
|
||||
|
||||
---
|
||||
|
||||
## Fallback Strategy & Unknown Devices
|
||||
|
||||
### Confidence Thresholds
|
||||
|
||||
```javascript
|
||||
function interpretConfidence(max_confidence, predictions) {
|
||||
if (max_confidence >= 0.85) {
|
||||
return {
|
||||
decision: 'auto_identify',
|
||||
message: 'High confidence match',
|
||||
show_alternatives: false
|
||||
};
|
||||
} else if (max_confidence >= 0.65) {
|
||||
return {
|
||||
decision: 'suggest',
|
||||
message: 'Likely match - please verify',
|
||||
show_alternatives: true,
|
||||
top_k: 3
|
||||
};
|
||||
} else if (max_confidence >= 0.40) {
|
||||
return {
|
||||
decision: 'multiple_options',
|
||||
message: 'Multiple possibilities detected',
|
||||
show_alternatives: true,
|
||||
top_k: 5
|
||||
};
|
||||
} else {
|
||||
return {
|
||||
decision: 'unknown',
|
||||
message: 'Unknown device - please help us identify',
|
||||
show_manual_form: true
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Unknown Device Handling
|
||||
|
||||
**UI Flow:**
|
||||
1. Show "Unknown Device" badge
|
||||
2. Display extracted features (frequency, protocol, etc.)
|
||||
3. Provide search box to find similar devices in database
|
||||
4. Allow manual device creation with form:
|
||||
- Manufacturer (text input with autocomplete)
|
||||
- Model (text input)
|
||||
- Device type (dropdown: garage door, weather sensor, etc.)
|
||||
- Notes (textarea)
|
||||
- Upload photo (optional)
|
||||
|
||||
**Backend:**
|
||||
- Store in `manual_identifications` table
|
||||
- Flag for community review
|
||||
- After 3+ confirmations, add to training set
|
||||
- Retrain model in next batch
|
||||
|
||||
---
|
||||
|
||||
## Privacy & Security Considerations
|
||||
|
||||
### Browser-Based ML Benefits
|
||||
|
||||
1. **Data Privacy:** Raw .sub files processed locally, not sent to server until user confirms
|
||||
2. **Bandwidth:** Only send metadata + matched device ID, not full signal
|
||||
3. **Offline Mode:** Model cached, works without internet
|
||||
|
||||
### Model Security
|
||||
|
||||
**Challenges:**
|
||||
- Models are public (JavaScript), can be reverse-engineered
|
||||
- Adversarial attacks possible (craft .sub files to fool classifier)
|
||||
|
||||
**Mitigations:**
|
||||
1. **Ensemble Defense:** Hard to fool all 3 classifiers simultaneously
|
||||
2. **Server-Side Validation:** Re-run classification on server, flag large discrepancies
|
||||
3. **Rate Limiting:** Prevent automated adversarial search
|
||||
4. **Community Verification:** Human oversight on all auto-IDs
|
||||
|
||||
---
|
||||
|
||||
## Implementation Roadmap
|
||||
|
||||
### Phase 1: Training Infrastructure (Week 1-2)
|
||||
|
||||
- [ ] Dataset labeling pipeline (automatic + file path extraction)
|
||||
- [ ] Feature extraction code (statistical features)
|
||||
- [ ] CNN training script (TensorFlow/Keras)
|
||||
- [ ] Statistical classifier training (LightGBM)
|
||||
- [ ] Model evaluation suite
|
||||
- [ ] Export to TensorFlow.js and ONNX
|
||||
|
||||
### Phase 2: Browser Integration (Week 3)
|
||||
|
||||
- [ ] JavaScript .sub parser (port from Python)
|
||||
- [ ] TensorFlow.js inference pipeline
|
||||
- [ ] ONNX Runtime Web integration
|
||||
- [ ] Ensemble voting logic
|
||||
- [ ] Web Worker for async processing
|
||||
- [ ] Model caching and versioning
|
||||
|
||||
### Phase 3: UI/UX (Week 4)
|
||||
|
||||
- [ ] Upload form with real-time prediction
|
||||
- [ ] Confidence visualization (progress bars, badges)
|
||||
- [ ] Manual labeling interface
|
||||
- [ ] Device search and selection
|
||||
- [ ] Photo upload for unknown devices
|
||||
|
||||
### Phase 4: Production Pipeline (Week 5-6)
|
||||
|
||||
- [ ] Automatic matching on server after upload
|
||||
- [ ] Background job queue (Celery/RQ)
|
||||
- [ ] Store ensemble results in `capture_matches`
|
||||
- [ ] API endpoints for match queries
|
||||
- [ ] Community verification system
|
||||
|
||||
### Phase 5: Continuous Learning (Ongoing)
|
||||
|
||||
- [ ] Weekly retraining pipeline
|
||||
- [ ] A/B testing framework
|
||||
- [ ] Model performance monitoring
|
||||
- [ ] Active learning (select most informative samples for labeling)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Short-Term (3 Months)
|
||||
|
||||
- [ ] Deploy v1.0 models (CNN + Statistical + Heuristic ensemble)
|
||||
- [ ] Achieve 75% top-1 accuracy on test set
|
||||
- [ ] Achieve 90% top-5 accuracy on test set
|
||||
- [ ] <300ms browser inference latency
|
||||
- [ ] 1,000+ manual labels collected
|
||||
- [ ] 60% auto-accept rate (users confirm without editing)
|
||||
|
||||
### Long-Term (12 Months)
|
||||
|
||||
- [ ] 85% top-1 accuracy (adding 10,000+ manual labels)
|
||||
- [ ] 95% top-5 accuracy
|
||||
- [ ] Support 200+ device classes
|
||||
- [ ] 75% auto-accept rate
|
||||
- [ ] Active learning pipeline reduces manual labeling by 50%
|
||||
- [ ] Mobile app integration (TensorFlow Lite)
|
||||
|
||||
---
|
||||
|
||||
## Alternative Approaches Considered
|
||||
|
||||
### Transformer-Based Models
|
||||
|
||||
**Pros:**
|
||||
- State-of-the-art for sequence data
|
||||
- Attention mechanism captures long-range dependencies
|
||||
|
||||
**Cons:**
|
||||
- Large model size (>10MB, too big for browser)
|
||||
- Slower inference (100ms+ on desktop)
|
||||
- Requires more training data (100k+ samples)
|
||||
|
||||
**Decision:** Revisit when dataset grows to 50k+ samples
|
||||
|
||||
### Autoencoder + Clustering
|
||||
|
||||
**Pros:**
|
||||
- Unsupervised, doesn't require labels
|
||||
- Can discover new device types automatically
|
||||
|
||||
**Cons:**
|
||||
- No direct classification, requires post-processing
|
||||
- Cluster assignment unstable with new data
|
||||
|
||||
**Decision:** Use for exploratory analysis, not production
|
||||
|
||||
### On-Device Training (Federated Learning)
|
||||
|
||||
**Pros:**
|
||||
- Users contribute to model without sharing data
|
||||
- Personalized models (local RF environment)
|
||||
|
||||
**Cons:**
|
||||
- Complex infrastructure (TensorFlow Federated)
|
||||
- Slow convergence, communication overhead
|
||||
|
||||
**Decision:** Future enhancement (Phase 2, 12+ months)
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The proposed hybrid ensemble system balances:
|
||||
- **Accuracy:** Multi-model approach covers diverse signal types
|
||||
- **Speed:** Browser-based inference, <300ms latency
|
||||
- **Deployability:** TensorFlow.js + ONNX, standard web stack
|
||||
- **Scalability:** Continuous learning from user feedback
|
||||
|
||||
**Next Steps:**
|
||||
1. Label initial 5,000 samples
|
||||
2. Train baseline models
|
||||
3. Implement browser inference pipeline
|
||||
4. Deploy v1.0 and collect production data
|
||||
|
||||
**Key Innovation:**
|
||||
Combining traditional signal processing (heuristics), statistical ML (interpretable features), and deep learning (CNN) in a weighted ensemble provides robustness and explainability while achieving high accuracy.
|
||||
Reference in New Issue
Block a user