giglez/docs/ML_DESIGN.md

# ML/Neural Network Design for RF Signal Classification

## Executive Summary

This document outlines the machine learning approach for automatic RF device identification in GigLez, focusing on Sub-GHz IoT signals (300-928 MHz). The design prioritizes practical deployment using JavaScript/TensorFlow.js for browser-based inference, with Python training pipeline.

---

## Research Foundation

### Key Papers & Approaches

1. **Deep Learning for RF Signal Classification** (Shi & Davaslioglu, 2019)
   - CNN architectures for modulation recognition
   - Achieves >95% accuracy above 2dB SNR
   - Uses IQ sample input (128-256 samples)

2. **RF Fingerprinting for IoT** (Jian et al., 2020)
   - Device-specific transmitter signatures
   - Massive experimental study on real IoT devices
   - Addresses resource-constrained edge deployment

3. **Practical RF Machine Learning** (Panoradio SDR)
   - CNN + RNN hybrid architectures
   - Categorical cross-entropy optimization
   - Real-world performance validation

### State-of-the-Art Insights

**What Works:**
- CNNs excel at extracting spatial features from signal representations
- 1D CNNs on raw pulse data perform well for timing-based protocols
- 2D CNNs on spectrograms capture frequency-domain patterns
- Ensemble methods (CNN + heuristics) outperform pure ML

**Challenges:**
- Single-transmission captures (Flipper Zero) vs. continuous streams (RTL-SDR)
- Variable signal lengths require padding or dynamic architectures
- Class imbalance (1000+ garage openers, <10 weather sensors)
- Synthetic vs. real-world signal distribution shift

---

## GigLez-Specific Requirements

### Input Characteristics

**Signal Types:**
1. **Decoded (.sub KEY format)**
   - Protocol, frequency, bit length, key data, timing element
   - Structured metadata, high information density
   - Best for heuristic matching (current system)

2. **RAW (.sub RAW format)**
   - Pulse/gap timing arrays (e.g., `2980 -240 520 -980 ...`)
   - Variable length (10-10,000+ samples)
   - Target for ML approach

3. **BinRAW (.sub BinRAW format)**
   - Binary pulse data
   - Less common, lower priority

**Challenge:** 97% of Flipper Zero captures are single-transmission (1-3 repeats), not continuous streams.

### Output Requirements

**Device Classification:**
- Primary: Device type (e.g., "Garage Door Opener", "Weather Sensor - Temperature")
- Secondary: Manufacturer (e.g., "Chamberlain", "Oregon Scientific")
- Tertiary: Model (e.g., "LiftMaster 891LM", "THGN123N")

**Confidence Scoring:**
- Probabilistic output (softmax)
- Combine with heuristic matcher scores
- Threshold: >0.7 for auto-ID, 0.5-0.7 for suggestions, <0.5 unknown

---

## Proposed Architecture: Hybrid Ensemble System

### Overview

Combine **three parallel classifiers** with weighted ensemble:

```
RAW .sub file
    ↓
[Feature Extraction]
    ├─→ 1D CNN (Temporal Features)       [35% weight]
    ├─→ Statistical Feature Classifier   [25% weight]
    └─→ Heuristic Matcher (existing)     [40% weight]
    ↓
[Weighted Ensemble]
    ↓
Device ID + Confidence
```

**Rationale:**
- Heuristic matcher handles decoded signals perfectly (1.0 confidence)
- CNN learns patterns in RAW signals the heuristics miss
- Statistical features provide interpretable fallback
- Weighted ensemble prevents over-reliance on any single method

---

## Model 1: 1D CNN for Pulse Timing Classification

### Architecture

**Input:**
- RAW pulse data, fixed length 512 samples (padded/truncated)
- Normalization: Divide by max(abs(value)) to range [-1, 1]
- Shape: `(batch_size, 512, 1)`

**Network Structure:**

```python
# TensorFlow.js equivalent structure
model = Sequential([
    # Block 1: Initial feature extraction
    Conv1D(64, kernel_size=16, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling1D(pool_size=4),
    Dropout(0.2),

    # Block 2: Mid-level features
    Conv1D(128, kernel_size=8, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling1D(pool_size=4),
    Dropout(0.3),

    # Block 3: High-level features
    Conv1D(256, kernel_size=4, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    GlobalAveragePooling1D(),

    # Classification head
    Dense(512, activation='relu'),
    Dropout(0.4),
    Dense(num_classes, activation='softmax')
])
```

**Hyperparameters:**
- Optimizer: Adam (lr=0.001, decay=1e-6)
- Loss: Categorical cross-entropy with label smoothing (0.1)
- Batch size: 32
- Epochs: 50 with early stopping (patience=5)

**Data Augmentation:**
- Random time stretching (0.9x - 1.1x)
- Gaussian noise injection (SNR 20-40 dB)
- Random clipping (simulate distance variations)

**Output:**
- Softmax probabilities for top 100 device classes
- Classes determined by dataset frequency (>50 examples)

---

## Model 2: Statistical Feature Classifier

### Feature Extraction (Handcrafted)

From RAW pulse data, extract 47 features:

**Timing Features (16):**
- Mean, median, std, min, max of positive pulses
- Mean, median, std, min, max of negative gaps
- Pulse/gap ratio, duty cycle
- Short pulse width, long pulse width (K-means clusters)
- Coefficient of variation for pulses and gaps

**Frequency Domain (12):**
- FFT magnitude peaks (top 5 frequencies)
- Dominant frequency, total energy, spectral centroid
- Bandwidth (95% energy containment)

**Pattern Features (10):**
- Autocorrelation peaks (indicates repeating patterns)
- Zero-crossing rate
- Peak count, valley count
- Longest run of similar-width pulses
- Entropy of pulse width distribution

**Metadata Features (9):**
- Frequency (normalized to 433/868/315/915 bands)
- Modulation type (one-hot: OOK, FSK, ASK)
- Total pulse count
- Total duration (ms)
- Average bit rate (estimated)

### Classifier

**Algorithm:** Gradient Boosted Trees (LightGBM or XGBoost)

**Rationale:**
- Handles tabular features excellently
- Fast inference (critical for browser deployment)
- Interpretable feature importance
- Robust to missing values

**Training:**
```python
import lightgbm as lgb

params = {
    'objective': 'multiclass',
    'num_class': num_classes,
    'metric': 'multi_logloss',
    'boosting_type': 'gbdt',
    'num_leaves': 63,
    'learning_rate': 0.05,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5,
    'max_depth': 8
}

model = lgb.train(params, train_data, num_boost_round=500)
```

**Export:** Convert to ONNX for browser inference via onnxruntime-web

---

## Model 3: Heuristic Matcher (Existing System)

**Keep current implementation** from `src/matcher/`:
- ExactMatcher, PartialMatcher, PatternMatcher, etc.
- Confidence scores: 0.5-1.0
- Fast, deterministic, handles decoded signals

**Integration:** Run in parallel with ML models, ensemble at end

---

## Ensemble Strategy

### Weighted Voting

```javascript
function ensemblePredict(cnn_probs, stat_probs, heuristic_results) {
    const weights = {
        cnn: 0.35,
        statistical: 0.25,
        heuristic: 0.40
    };

    // Combine probabilities
    let combined = {};

    // CNN contribution
    for (let [device_id, prob] of Object.entries(cnn_probs)) {
        combined[device_id] = (combined[device_id] || 0) + prob * weights.cnn;
    }

    // Statistical contribution
    for (let [device_id, prob] of Object.entries(stat_probs)) {
        combined[device_id] = (combined[device_id] || 0) + prob * weights.statistical;
    }

    // Heuristic contribution (convert confidence to probability)
    for (let match of heuristic_results) {
        let device_id = match.device_id;
        let prob = match.confidence;  // Already 0-1 range
        combined[device_id] = (combined[device_id] || 0) + prob * weights.heuristic;
    }

    // Sort by combined score
    let sorted = Object.entries(combined)
        .sort((a, b) => b[1] - a[1])
        .slice(0, 10);  // Top 10

    return sorted.map(([device_id, score]) => ({
        device_id: device_id,
        confidence: score,
        method: 'ensemble'
    }));
}
```

### Dynamic Weighting

Adjust weights based on input signal type:

| Signal Type | CNN Weight | Statistical Weight | Heuristic Weight |
|-------------|------------|--------------------| -----------------|
| Decoded (KEY format) | 0.10 | 0.10 | 0.80 |
| RAW with >500 samples | 0.45 | 0.25 | 0.30 |
| RAW with <100 samples | 0.20 | 0.40 | 0.40 |
| Known protocol detected | 0.05 | 0.05 | 0.90 |

---

## Training Data Pipeline

### Dataset Structure

**Source:** Downloaded datasets (41,334 .sub files)

**Labeling Strategy:**

1. **Automatic Labels (High Confidence)**
   - Use file path structure (e.g., `Weather_stations/Oregon_Scientific/...`)
   - Decoded protocol name match
   - RTL_433 successful decode
   - **Estimated:** 15,000 high-quality labels

2. **Manual Labels (User-Submitted)**
   - Upload form includes device selection
   - Community verification voting
   - Store in `manual_identifications` table
   - **Expected:** 1,000+ labels over 6 months

3. **Semi-Supervised Learning**
   - Use high-confidence heuristic matches (1.0 confidence) as pseudo-labels
   - Iterative refinement: Train model → Generate predictions → Review low-confidence → Retrain
   - **Potential:** 25,000+ samples

### Class Distribution Handling

**Problem:** Long-tail distribution
- Top 10 classes: 60% of data (garage door openers)
- Bottom 50 classes: 5% of data (weather sensors, TPMS)

**Solutions:**
1. **Class Weighting:**
   ```python
   from sklearn.utils.class_weight import compute_class_weight

   class_weights = compute_class_weight(
       'balanced',
       classes=np.unique(y_train),
       y=y_train
   )
   ```

2. **Focal Loss:** Penalize easy examples (common classes)
   ```python
   def focal_loss(y_true, y_pred, alpha=0.25, gamma=2.0):
       ce = -y_true * np.log(y_pred + 1e-7)
       weight = alpha * y_true * np.power(1 - y_pred, gamma)
       return np.sum(weight * ce)
   ```

3. **Stratified Sampling:** Ensure rare classes in each batch

### Train/Val/Test Split

```python
from sklearn.model_selection import StratifiedShuffleSplit

splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y))

# Further split training into train/val
splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.15, random_state=42)
train_idx, val_idx = next(splitter.split(X[train_idx], y[train_idx]))
```

**Split:**
- Training: 68% (~28,000 samples)
- Validation: 12% (~5,000 samples)
- Test: 20% (~8,000 samples)

---

## JavaScript Deployment (TensorFlow.js)

### Browser-Based Inference

**Why Browser:**
- Instant feedback during upload
- No server cost for inference
- Privacy: .sub files never leave device for pre-screening
- Offline capability (PWA)

### Model Conversion

**1. Train in Python (TensorFlow/Keras)**
```python
model.save('models/cnn_classifier_v1.h5')
```

**2. Convert to TensorFlow.js**
```bash
tensorflowjs_converter \
    --input_format=keras \
    --output_format=tfjs_graph_model \
    models/cnn_classifier_v1.h5 \
    static/models/cnn_v1/
```

**3. Load in Browser**
```javascript
import * as tf from '@tensorflow/tfjs';

const model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');

// Inference
function predict(rawPulseArray) {
    // Preprocess
    let normalized = rawPulseArray.map(x => x / Math.max(...rawPulseArray.map(Math.abs)));
    let padded = padOrTruncate(normalized, 512);

    // Convert to tensor
    let tensor = tf.tensor3d([padded.map(x => [x])], [1, 512, 1]);

    // Predict
    let probs = model.predict(tensor);
    let probsArray = await probs.data();

    // Get top 5
    let indexed = Array.from(probsArray).map((p, i) => [i, p]);
    indexed.sort((a, b) => b[1] - a[1]);

    return indexed.slice(0, 5).map(([idx, prob]) => ({
        device_id: classMapping[idx],
        confidence: prob
    }));
}
```

### Performance Optimization

**Model Quantization:**
```bash
tensorflowjs_converter \
    --input_format=keras \
    --output_format=tfjs_graph_model \
    --quantization_bytes=2 \
    models/cnn_classifier_v1.h5 \
    static/models/cnn_v1_quantized/
```

**Benefits:**
- 4x smaller model (16-bit weights vs 32-bit)
- 2x faster inference on mobile
- Minimal accuracy loss (<1%)

**Web Workers:**
```javascript
// Main thread
const worker = new Worker('/static/js/ml_worker.js');

worker.postMessage({
    type: 'predict',
    raw_data: pulseArray
});

worker.onmessage = (e) => {
    console.log('Predictions:', e.data.predictions);
};

// Worker thread (ml_worker.js)
importScripts('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs');

let model = null;

self.onmessage = async (e) => {
    if (!model) {
        model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');
    }

    const predictions = predict(e.data.raw_data);
    self.postMessage({ predictions });
};
```

---

## Statistical Feature Classifier Deployment

### ONNX Runtime Web

**1. Train in Python (LightGBM)**
```python
import lightgbm as lgb
import onnxmltools
from onnxmltools.convert import convert_lightgbm

# Train model
model = lgb.train(params, train_data)

# Convert to ONNX
onnx_model = convert_lightgbm(
    model,
    initial_types=[('input', FloatTensorType([None, 47]))],
    target_opset=12
)

onnxmltools.utils.save_model(onnx_model, 'models/stat_classifier_v1.onnx')
```

**2. Load in Browser**
```javascript
import * as ort from 'onnxruntime-web';

const session = await ort.InferenceSession.create('/static/models/stat_v1.onnx');

async function predictStatistical(features) {
    // features: Float32Array of length 47
    const tensor = new ort.Tensor('float32', features, [1, 47]);
    const feeds = { input: tensor };
    const results = await session.run(feeds);

    // results.probabilities: [1, num_classes]
    const probs = results.probabilities.data;

    return Array.from(probs).map((prob, idx) => ({
        device_id: classMapping[idx],
        confidence: prob
    })).sort((a, b) => b.confidence - a.confidence).slice(0, 5);
}
```

---

## Training Schedule & Versioning

### Initial Training (Week 1)

**Goal:** Establish baseline

1. Label 5,000 high-confidence samples (automatic + file paths)
2. Train CNN on 50 most common classes
3. Train statistical classifier on same 50 classes
4. Export to TensorFlow.js and ONNX
5. Deploy v1.0 models

**Metrics:**
- Top-1 accuracy: Target >70%
- Top-5 accuracy: Target >85%
- Inference time: <200ms on desktop, <500ms mobile

### Continuous Improvement (Ongoing)

**Weekly Retraining:**
1. Collect new manual labels from upload form
2. Review confidence scores on production data
3. Add low-confidence samples to manual review queue
4. Retrain models with expanded dataset
5. A/B test new model vs. current model
6. Deploy if metrics improve by >2%

**Version Tracking:**
```
models/
├── cnn/
│   ├── v1.0_2026-01-15/
│   │   ├── model.json
│   │   ├── group1-shard1of2.bin
│   │   ├── group1-shard2of2.bin
│   │   ├── metadata.json  (accuracy, classes, date)
│   │   └── training_log.txt
│   ├── v1.1_2026-01-22/
│   └── latest → v1.1_2026-01-22
├── statistical/
│   ├── v1.0_2026-01-15/
│   │   ├── model.onnx
│   │   ├── feature_metadata.json
│   │   └── training_log.txt
│   └── latest → v1.0_2026-01-15
└── ensemble_config.json
```

---

## Evaluation Metrics

### Model Performance

**Classification Metrics:**
- Top-1 Accuracy
- Top-5 Accuracy
- Macro-averaged F1 (handle class imbalance)
- Per-class Precision/Recall (focus on rare classes)

**Confidence Calibration:**
- Expected Calibration Error (ECE)
- Reliability diagram (predicted prob vs. actual accuracy)

**Ensemble Performance:**
- Improvement over best single model
- Disagreement analysis (where models differ)

### Production Metrics

**User Behavior:**
- Auto-accept rate (user confirms ML prediction without editing)
- Manual correction rate
- "Unknown device" submission rate

**System Performance:**
- Inference latency (p50, p95, p99)
- Model download size
- Cache hit rate

**Data Quality:**
- Label confidence distribution
- Inter-annotator agreement (when multiple users label same capture)

---

## Fallback Strategy & Unknown Devices

### Confidence Thresholds

```javascript
function interpretConfidence(max_confidence, predictions) {
    if (max_confidence >= 0.85) {
        return {
            decision: 'auto_identify',
            message: 'High confidence match',
            show_alternatives: false
        };
    } else if (max_confidence >= 0.65) {
        return {
            decision: 'suggest',
            message: 'Likely match - please verify',
            show_alternatives: true,
            top_k: 3
        };
    } else if (max_confidence >= 0.40) {
        return {
            decision: 'multiple_options',
            message: 'Multiple possibilities detected',
            show_alternatives: true,
            top_k: 5
        };
    } else {
        return {
            decision: 'unknown',
            message: 'Unknown device - please help us identify',
            show_manual_form: true
        };
    }
}
```

### Unknown Device Handling

**UI Flow:**
1. Show "Unknown Device" badge
2. Display extracted features (frequency, protocol, etc.)
3. Provide search box to find similar devices in database
4. Allow manual device creation with form:
   - Manufacturer (text input with autocomplete)
   - Model (text input)
   - Device type (dropdown: garage door, weather sensor, etc.)
   - Notes (textarea)
   - Upload photo (optional)

**Backend:**
- Store in `manual_identifications` table
- Flag for community review
- After 3+ confirmations, add to training set
- Retrain model in next batch

---

## Privacy & Security Considerations

### Browser-Based ML Benefits

1. **Data Privacy:** Raw .sub files processed locally, not sent to server until user confirms
2. **Bandwidth:** Only send metadata + matched device ID, not full signal
3. **Offline Mode:** Model cached, works without internet

### Model Security

**Challenges:**
- Models are public (JavaScript), can be reverse-engineered
- Adversarial attacks possible (craft .sub files to fool classifier)

**Mitigations:**
1. **Ensemble Defense:** Hard to fool all 3 classifiers simultaneously
2. **Server-Side Validation:** Re-run classification on server, flag large discrepancies
3. **Rate Limiting:** Prevent automated adversarial search
4. **Community Verification:** Human oversight on all auto-IDs

---

## Implementation Roadmap

### Phase 1: Training Infrastructure (Week 1-2)

- [ ] Dataset labeling pipeline (automatic + file path extraction)
- [ ] Feature extraction code (statistical features)
- [ ] CNN training script (TensorFlow/Keras)
- [ ] Statistical classifier training (LightGBM)
- [ ] Model evaluation suite
- [ ] Export to TensorFlow.js and ONNX

### Phase 2: Browser Integration (Week 3)

- [ ] JavaScript .sub parser (port from Python)
- [ ] TensorFlow.js inference pipeline
- [ ] ONNX Runtime Web integration
- [ ] Ensemble voting logic
- [ ] Web Worker for async processing
- [ ] Model caching and versioning

### Phase 3: UI/UX (Week 4)

- [ ] Upload form with real-time prediction
- [ ] Confidence visualization (progress bars, badges)
- [ ] Manual labeling interface
- [ ] Device search and selection
- [ ] Photo upload for unknown devices

### Phase 4: Production Pipeline (Week 5-6)

- [ ] Automatic matching on server after upload
- [ ] Background job queue (Celery/RQ)
- [ ] Store ensemble results in `capture_matches`
- [ ] API endpoints for match queries
- [ ] Community verification system

### Phase 5: Continuous Learning (Ongoing)

- [ ] Weekly retraining pipeline
- [ ] A/B testing framework
- [ ] Model performance monitoring
- [ ] Active learning (select most informative samples for labeling)

---

## Success Criteria

### Short-Term (3 Months)

- [ ] Deploy v1.0 models (CNN + Statistical + Heuristic ensemble)
- [ ] Achieve 75% top-1 accuracy on test set
- [ ] Achieve 90% top-5 accuracy on test set
- [ ] <300ms browser inference latency
- [ ] 1,000+ manual labels collected
- [ ] 60% auto-accept rate (users confirm without editing)

### Long-Term (12 Months)

- [ ] 85% top-1 accuracy (adding 10,000+ manual labels)
- [ ] 95% top-5 accuracy
- [ ] Support 200+ device classes
- [ ] 75% auto-accept rate
- [ ] Active learning pipeline reduces manual labeling by 50%
- [ ] Mobile app integration (TensorFlow Lite)

---

## Alternative Approaches Considered

### Transformer-Based Models

**Pros:**
- State-of-the-art for sequence data
- Attention mechanism captures long-range dependencies

**Cons:**
- Large model size (>10MB, too big for browser)
- Slower inference (100ms+ on desktop)
- Requires more training data (100k+ samples)

**Decision:** Revisit when dataset grows to 50k+ samples

### Autoencoder + Clustering

**Pros:**
- Unsupervised, doesn't require labels
- Can discover new device types automatically

**Cons:**
- No direct classification, requires post-processing
- Cluster assignment unstable with new data

**Decision:** Use for exploratory analysis, not production

### On-Device Training (Federated Learning)

**Pros:**
- Users contribute to model without sharing data
- Personalized models (local RF environment)

**Cons:**
- Complex infrastructure (TensorFlow Federated)
- Slow convergence, communication overhead

**Decision:** Future enhancement (Phase 2, 12+ months)

---

## Conclusion

The proposed hybrid ensemble system balances:
- **Accuracy:** Multi-model approach covers diverse signal types
- **Speed:** Browser-based inference, <300ms latency
- **Deployability:** TensorFlow.js + ONNX, standard web stack
- **Scalability:** Continuous learning from user feedback

**Next Steps:**
1. Label initial 5,000 samples
2. Train baseline models
3. Implement browser inference pipeline
4. Deploy v1.0 and collect production data

**Key Innovation:**
Combining traditional signal processing (heuristics), statistical ML (interpretable features), and deep learning (CNN) in a weighted ensemble provides robustness and explainability while achieving high accuracy.