giglez/docs/ML_DESIGN.md

ML/Neural Network Design for RF Signal Classification

Executive Summary

This document outlines the machine learning approach for automatic RF device identification in GigLez, focusing on Sub-GHz IoT signals (300-928 MHz). The design prioritizes practical deployment using JavaScript/TensorFlow.js for browser-based inference, with Python training pipeline.

---

Research Foundation

Key Papers & Approaches

1. Deep Learning for RF Signal Classification (Shi & Davaslioglu, 2019)

   • CNN architectures for modulation recognition
   • Achieves >95% accuracy above 2dB SNR
   • Uses IQ sample input (128-256 samples)

2. RF Fingerprinting for IoT (Jian et al., 2020)

   • Device-specific transmitter signatures
   • Massive experimental study on real IoT devices
   • Addresses resource-constrained edge deployment

3. Practical RF Machine Learning (Panoradio SDR)

   • CNN + RNN hybrid architectures
   • Categorical cross-entropy optimization
   • Real-world performance validation

State-of-the-Art Insights

What Works:

• CNNs excel at extracting spatial features from signal representations
• 1D CNNs on raw pulse data perform well for timing-based protocols
• 2D CNNs on spectrograms capture frequency-domain patterns
• Ensemble methods (CNN + heuristics) outperform pure ML

Challenges:

• Single-transmission captures (Flipper Zero) vs. continuous streams (RTL-SDR)
• Variable signal lengths require padding or dynamic architectures
• Class imbalance (1000+ garage openers, <10 weather sensors)
• Synthetic vs. real-world signal distribution shift

---

GigLez-Specific Requirements

Input Characteristics

Signal Types:

1. Decoded (.sub KEY format)

   • Protocol, frequency, bit length, key data, timing element
   • Structured metadata, high information density
   • Best for heuristic matching (current system)

2. RAW (.sub RAW format)

   • Pulse/gap timing arrays (e.g., 2980 -240 520 -980 ...)
   • Variable length (10-10,000+ samples)
   • Target for ML approach

3. BinRAW (.sub BinRAW format)

   • Binary pulse data
   • Less common, lower priority

Challenge: 97% of Flipper Zero captures are single-transmission (1-3 repeats), not continuous streams.

Output Requirements

Device Classification:

• Primary: Device type (e.g., "Garage Door Opener", "Weather Sensor - Temperature")
• Secondary: Manufacturer (e.g., "Chamberlain", "Oregon Scientific")
• Tertiary: Model (e.g., "LiftMaster 891LM", "THGN123N")

Confidence Scoring:

• Probabilistic output (softmax)
• Combine with heuristic matcher scores
• Threshold: >0.7 for auto-ID, 0.5-0.7 for suggestions, <0.5 unknown

---

Proposed Architecture: Hybrid Ensemble System

Overview

Combine three parallel classifiers with weighted ensemble:

RAW .sub file
    ↓
[Feature Extraction]
    ├─→ 1D CNN (Temporal Features)       [35% weight]
    ├─→ Statistical Feature Classifier   [25% weight]
    └─→ Heuristic Matcher (existing)     [40% weight]
    ↓
[Weighted Ensemble]
    ↓
Device ID + Confidence

Rationale:

• Heuristic matcher handles decoded signals perfectly (1.0 confidence)
• CNN learns patterns in RAW signals the heuristics miss
• Statistical features provide interpretable fallback
• Weighted ensemble prevents over-reliance on any single method

---

Model 1: 1D CNN for Pulse Timing Classification

Architecture

Input:

• RAW pulse data, fixed length 512 samples (padded/truncated)
• Normalization: Divide by max(abs(value)) to range [-1, 1]
• Shape: (batch_size, 512, 1)

Network Structure:

# TensorFlow.js equivalent structure
model = Sequential([
    # Block 1: Initial feature extraction
    Conv1D(64, kernel_size=16, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling1D(pool_size=4),
    Dropout(0.2),

    # Block 2: Mid-level features
    Conv1D(128, kernel_size=8, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling1D(pool_size=4),
    Dropout(0.3),

    # Block 3: High-level features
    Conv1D(256, kernel_size=4, strides=2, activation='relu', padding='same'),
    BatchNormalization(),
    GlobalAveragePooling1D(),

    # Classification head
    Dense(512, activation='relu'),
    Dropout(0.4),
    Dense(num_classes, activation='softmax')
])

Hyperparameters:

• Optimizer: Adam (lr=0.001, decay=1e-6)
• Loss: Categorical cross-entropy with label smoothing (0.1)
• Batch size: 32
• Epochs: 50 with early stopping (patience=5)

Data Augmentation:

• Random time stretching (0.9x - 1.1x)
• Gaussian noise injection (SNR 20-40 dB)
• Random clipping (simulate distance variations)

Output:

• Softmax probabilities for top 100 device classes
• Classes determined by dataset frequency (>50 examples)

---

Model 2: Statistical Feature Classifier

Feature Extraction (Handcrafted)

From RAW pulse data, extract 47 features:

Timing Features (16):

• Mean, median, std, min, max of positive pulses
• Mean, median, std, min, max of negative gaps
• Pulse/gap ratio, duty cycle
• Short pulse width, long pulse width (K-means clusters)
• Coefficient of variation for pulses and gaps

Frequency Domain (12):

• FFT magnitude peaks (top 5 frequencies)
• Dominant frequency, total energy, spectral centroid
• Bandwidth (95% energy containment)

Pattern Features (10):

• Autocorrelation peaks (indicates repeating patterns)
• Zero-crossing rate
• Peak count, valley count
• Longest run of similar-width pulses
• Entropy of pulse width distribution

Metadata Features (9):

• Frequency (normalized to 433/868/315/915 bands)
• Modulation type (one-hot: OOK, FSK, ASK)
• Total pulse count
• Total duration (ms)
• Average bit rate (estimated)

Classifier

Algorithm: Gradient Boosted Trees (LightGBM or XGBoost)

Rationale:

• Handles tabular features excellently
• Fast inference (critical for browser deployment)
• Interpretable feature importance
• Robust to missing values

Training:

import lightgbm as lgb

params = {
    'objective': 'multiclass',
    'num_class': num_classes,
    'metric': 'multi_logloss',
    'boosting_type': 'gbdt',
    'num_leaves': 63,
    'learning_rate': 0.05,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5,
    'max_depth': 8
}

model = lgb.train(params, train_data, num_boost_round=500)

Export: Convert to ONNX for browser inference via onnxruntime-web

---

Model 3: Heuristic Matcher (Existing System)

Keep current implementation from src/matcher/:

• ExactMatcher, PartialMatcher, PatternMatcher, etc.
• Confidence scores: 0.5-1.0
• Fast, deterministic, handles decoded signals

Integration: Run in parallel with ML models, ensemble at end

---

Ensemble Strategy

Weighted Voting

function ensemblePredict(cnn_probs, stat_probs, heuristic_results) {
    const weights = {
        cnn: 0.35,
        statistical: 0.25,
        heuristic: 0.40
    };

    // Combine probabilities
    let combined = {};

    // CNN contribution
    for (let [device_id, prob] of Object.entries(cnn_probs)) {
        combined[device_id] = (combined[device_id] || 0) + prob * weights.cnn;
    }

    // Statistical contribution
    for (let [device_id, prob] of Object.entries(stat_probs)) {
        combined[device_id] = (combined[device_id] || 0) + prob * weights.statistical;
    }

    // Heuristic contribution (convert confidence to probability)
    for (let match of heuristic_results) {
        let device_id = match.device_id;
        let prob = match.confidence;  // Already 0-1 range
        combined[device_id] = (combined[device_id] || 0) + prob * weights.heuristic;
    }

    // Sort by combined score
    let sorted = Object.entries(combined)
        .sort((a, b) => b[1] - a[1])
        .slice(0, 10);  // Top 10

    return sorted.map(([device_id, score]) => ({
        device_id: device_id,
        confidence: score,
        method: 'ensemble'
    }));
}

Dynamic Weighting

Adjust weights based on input signal type:

Signal Type	CNN Weight	Statistical Weight	Heuristic Weight
Decoded (KEY format)	0.10	0.10	0.80
RAW with >500 samples	0.45	0.25	0.30
RAW with <100 samples	0.20	0.40	0.40
Known protocol detected	0.05	0.05	0.90

---

Training Data Pipeline

Dataset Structure

Source: Downloaded datasets (41,334 .sub files)

Labeling Strategy:

1. Automatic Labels (High Confidence)

   • Use file path structure (e.g., Weather_stations/Oregon_Scientific/...)
   • Decoded protocol name match
   • RTL_433 successful decode
   • Estimated: 15,000 high-quality labels

2. Manual Labels (User-Submitted)

   • Upload form includes device selection
   • Community verification voting
   • Store in manual_identifications table
   • Expected: 1,000+ labels over 6 months

3. Semi-Supervised Learning

   • Use high-confidence heuristic matches (1.0 confidence) as pseudo-labels
   • Iterative refinement: Train model → Generate predictions → Review low-confidence → Retrain
   • Potential: 25,000+ samples

Class Distribution Handling

Problem: Long-tail distribution

• Top 10 classes: 60% of data (garage door openers)
• Bottom 50 classes: 5% of data (weather sensors, TPMS)

Solutions:

1. Class Weighting:

   from sklearn.utils.class_weight import compute_class_weight
   
   class_weights = compute_class_weight(
       'balanced',
       classes=np.unique(y_train),
       y=y_train
   )
   

2. Focal Loss: Penalize easy examples (common classes)

   def focal_loss(y_true, y_pred, alpha=0.25, gamma=2.0):
       ce = -y_true * np.log(y_pred + 1e-7)
       weight = alpha * y_true * np.power(1 - y_pred, gamma)
       return np.sum(weight * ce)
   

3. Stratified Sampling: Ensure rare classes in each batch

Train/Val/Test Split

from sklearn.model_selection import StratifiedShuffleSplit

splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y))

# Further split training into train/val
splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.15, random_state=42)
train_idx, val_idx = next(splitter.split(X[train_idx], y[train_idx]))

Split:

• Training: 68% (~28,000 samples)
• Validation: 12% (~5,000 samples)
• Test: 20% (~8,000 samples)

---

JavaScript Deployment (TensorFlow.js)

Browser-Based Inference

Why Browser:

• Instant feedback during upload
• No server cost for inference
• Privacy: .sub files never leave device for pre-screening
• Offline capability (PWA)

Model Conversion

1. Train in Python (TensorFlow/Keras)

model.save('models/cnn_classifier_v1.h5')

2. Convert to TensorFlow.js

tensorflowjs_converter \
    --input_format=keras \
    --output_format=tfjs_graph_model \
    models/cnn_classifier_v1.h5 \
    static/models/cnn_v1/

3. Load in Browser

import * as tf from '@tensorflow/tfjs';

const model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');

// Inference
function predict(rawPulseArray) {
    // Preprocess
    let normalized = rawPulseArray.map(x => x / Math.max(...rawPulseArray.map(Math.abs)));
    let padded = padOrTruncate(normalized, 512);

    // Convert to tensor
    let tensor = tf.tensor3d([padded.map(x => [x])], [1, 512, 1]);

    // Predict
    let probs = model.predict(tensor);
    let probsArray = await probs.data();

    // Get top 5
    let indexed = Array.from(probsArray).map((p, i) => [i, p]);
    indexed.sort((a, b) => b[1] - a[1]);

    return indexed.slice(0, 5).map(([idx, prob]) => ({
        device_id: classMapping[idx],
        confidence: prob
    }));
}

Performance Optimization

Model Quantization:

tensorflowjs_converter \
    --input_format=keras \
    --output_format=tfjs_graph_model \
    --quantization_bytes=2 \
    models/cnn_classifier_v1.h5 \
    static/models/cnn_v1_quantized/

Benefits:

• 4x smaller model (16-bit weights vs 32-bit)
• 2x faster inference on mobile
• Minimal accuracy loss (<1%)

Web Workers:

// Main thread
const worker = new Worker('/static/js/ml_worker.js');

worker.postMessage({
    type: 'predict',
    raw_data: pulseArray
});

worker.onmessage = (e) => {
    console.log('Predictions:', e.data.predictions);
};

// Worker thread (ml_worker.js)
importScripts('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs');

let model = null;

self.onmessage = async (e) => {
    if (!model) {
        model = await tf.loadGraphModel('/static/models/cnn_v1/model.json');
    }

    const predictions = predict(e.data.raw_data);
    self.postMessage({ predictions });
};

---

Statistical Feature Classifier Deployment

ONNX Runtime Web

1. Train in Python (LightGBM)

import lightgbm as lgb
import onnxmltools
from onnxmltools.convert import convert_lightgbm

# Train model
model = lgb.train(params, train_data)

# Convert to ONNX
onnx_model = convert_lightgbm(
    model,
    initial_types=[('input', FloatTensorType([None, 47]))],
    target_opset=12
)

onnxmltools.utils.save_model(onnx_model, 'models/stat_classifier_v1.onnx')

2. Load in Browser

import * as ort from 'onnxruntime-web';

const session = await ort.InferenceSession.create('/static/models/stat_v1.onnx');

async function predictStatistical(features) {
    // features: Float32Array of length 47
    const tensor = new ort.Tensor('float32', features, [1, 47]);
    const feeds = { input: tensor };
    const results = await session.run(feeds);

    // results.probabilities: [1, num_classes]
    const probs = results.probabilities.data;

    return Array.from(probs).map((prob, idx) => ({
        device_id: classMapping[idx],
        confidence: prob
    })).sort((a, b) => b.confidence - a.confidence).slice(0, 5);
}

---

Training Schedule & Versioning

Initial Training (Week 1)

Goal: Establish baseline

1. Label 5,000 high-confidence samples (automatic + file paths)
2. Train CNN on 50 most common classes
3. Train statistical classifier on same 50 classes
4. Export to TensorFlow.js and ONNX
5. Deploy v1.0 models

Metrics:

• Top-1 accuracy: Target >70%
• Top-5 accuracy: Target >85%
• Inference time: <200ms on desktop, <500ms mobile

Continuous Improvement (Ongoing)

Weekly Retraining:

1. Collect new manual labels from upload form
2. Review confidence scores on production data
3. Add low-confidence samples to manual review queue
4. Retrain models with expanded dataset
5. A/B test new model vs. current model
6. Deploy if metrics improve by >2%

Version Tracking:

models/
├── cnn/
│   ├── v1.0_2026-01-15/
│   │   ├── model.json
│   │   ├── group1-shard1of2.bin
│   │   ├── group1-shard2of2.bin
│   │   ├── metadata.json  (accuracy, classes, date)
│   │   └── training_log.txt
│   ├── v1.1_2026-01-22/
│   └── latest → v1.1_2026-01-22
├── statistical/
│   ├── v1.0_2026-01-15/
│   │   ├── model.onnx
│   │   ├── feature_metadata.json
│   │   └── training_log.txt
│   └── latest → v1.0_2026-01-15
└── ensemble_config.json

---

Evaluation Metrics

Model Performance

Classification Metrics:

• Top-1 Accuracy
• Top-5 Accuracy
• Macro-averaged F1 (handle class imbalance)
• Per-class Precision/Recall (focus on rare classes)

Confidence Calibration:

• Expected Calibration Error (ECE)
• Reliability diagram (predicted prob vs. actual accuracy)

Ensemble Performance:

• Improvement over best single model
• Disagreement analysis (where models differ)

Production Metrics

User Behavior:

• Auto-accept rate (user confirms ML prediction without editing)
• Manual correction rate
• "Unknown device" submission rate

System Performance:

• Inference latency (p50, p95, p99)
• Model download size
• Cache hit rate

Data Quality:

• Label confidence distribution
• Inter-annotator agreement (when multiple users label same capture)

---

Fallback Strategy & Unknown Devices

Confidence Thresholds

function interpretConfidence(max_confidence, predictions) {
    if (max_confidence >= 0.85) {
        return {
            decision: 'auto_identify',
            message: 'High confidence match',
            show_alternatives: false
        };
    } else if (max_confidence >= 0.65) {
        return {
            decision: 'suggest',
            message: 'Likely match - please verify',
            show_alternatives: true,
            top_k: 3
        };
    } else if (max_confidence >= 0.40) {
        return {
            decision: 'multiple_options',
            message: 'Multiple possibilities detected',
            show_alternatives: true,
            top_k: 5
        };
    } else {
        return {
            decision: 'unknown',
            message: 'Unknown device - please help us identify',
            show_manual_form: true
        };
    }
}

Unknown Device Handling

UI Flow:

1. Show "Unknown Device" badge
2. Display extracted features (frequency, protocol, etc.)
3. Provide search box to find similar devices in database
4. Allow manual device creation with form:
   • Manufacturer (text input with autocomplete)
   • Model (text input)
   • Device type (dropdown: garage door, weather sensor, etc.)
   • Notes (textarea)
   • Upload photo (optional)

Backend:

• Store in manual_identifications table
• Flag for community review
• After 3+ confirmations, add to training set
• Retrain model in next batch

---

Privacy & Security Considerations

Browser-Based ML Benefits

1. Data Privacy: Raw .sub files processed locally, not sent to server until user confirms
2. Bandwidth: Only send metadata + matched device ID, not full signal
3. Offline Mode: Model cached, works without internet

Model Security

Challenges:

• Models are public (JavaScript), can be reverse-engineered
• Adversarial attacks possible (craft .sub files to fool classifier)

Mitigations:

1. Ensemble Defense: Hard to fool all 3 classifiers simultaneously
2. Server-Side Validation: Re-run classification on server, flag large discrepancies
3. Rate Limiting: Prevent automated adversarial search
4. Community Verification: Human oversight on all auto-IDs

---

Implementation Roadmap

Phase 1: Training Infrastructure (Week 1-2)

• Dataset labeling pipeline (automatic + file path extraction)
• Feature extraction code (statistical features)
• CNN training script (TensorFlow/Keras)
• Statistical classifier training (LightGBM)
• Model evaluation suite
• Export to TensorFlow.js and ONNX

Phase 2: Browser Integration (Week 3)

• JavaScript .sub parser (port from Python)
• TensorFlow.js inference pipeline
• ONNX Runtime Web integration
• Ensemble voting logic
• Web Worker for async processing
• Model caching and versioning

Phase 3: UI/UX (Week 4)

• Upload form with real-time prediction
• Confidence visualization (progress bars, badges)
• Manual labeling interface
• Device search and selection
• Photo upload for unknown devices

Phase 4: Production Pipeline (Week 5-6)

• Automatic matching on server after upload
• Background job queue (Celery/RQ)
• Store ensemble results in capture_matches
• API endpoints for match queries
• Community verification system

Phase 5: Continuous Learning (Ongoing)

• Weekly retraining pipeline
• A/B testing framework
• Model performance monitoring
• Active learning (select most informative samples for labeling)

---

Success Criteria

Short-Term (3 Months)

• Deploy v1.0 models (CNN + Statistical + Heuristic ensemble)
• Achieve 75% top-1 accuracy on test set
• Achieve 90% top-5 accuracy on test set
• <300ms browser inference latency
• 1,000+ manual labels collected
• 60% auto-accept rate (users confirm without editing)

Long-Term (12 Months)

• 85% top-1 accuracy (adding 10,000+ manual labels)
• 95% top-5 accuracy
• Support 200+ device classes
• 75% auto-accept rate
• Active learning pipeline reduces manual labeling by 50%
• Mobile app integration (TensorFlow Lite)

---

Alternative Approaches Considered

Transformer-Based Models

Pros:

• State-of-the-art for sequence data
• Attention mechanism captures long-range dependencies

Cons:

• Large model size (>10MB, too big for browser)
• Slower inference (100ms+ on desktop)
• Requires more training data (100k+ samples)

Decision: Revisit when dataset grows to 50k+ samples

Autoencoder + Clustering

Pros:

• Unsupervised, doesn't require labels
• Can discover new device types automatically

Cons:

• No direct classification, requires post-processing
• Cluster assignment unstable with new data

Decision: Use for exploratory analysis, not production

On-Device Training (Federated Learning)

Pros:

• Users contribute to model without sharing data
• Personalized models (local RF environment)

Cons:

• Complex infrastructure (TensorFlow Federated)
• Slow convergence, communication overhead

Decision: Future enhancement (Phase 2, 12+ months)

---

Conclusion

The proposed hybrid ensemble system balances:

• Accuracy: Multi-model approach covers diverse signal types
• Speed: Browser-based inference, <300ms latency
• Deployability: TensorFlow.js + ONNX, standard web stack
• Scalability: Continuous learning from user feedback

Next Steps:

1. Label initial 5,000 samples
2. Train baseline models
3. Implement browser inference pipeline
4. Deploy v1.0 and collect production data

Key Innovation: Combining traditional signal processing (heuristics), statistical ML (interpretable features), and deep learning (CNN) in a weighted ensemble provides robustness and explainability while achieving high accuracy.