leetcrypt
a46af03f90
Trains a PyTorch 1D CNN on encoded RAW pulse arrays as the Phase 3B leg of the designed ensemble, scored on the SAME group-disjoint held-out test set as the heuristic and statistical models (GroupShuffleSplit by device sub-folder, no near-duplicate leakage). RESULT — CNN is data-starved and loses decisively: CNN balanced 0.338, top-1 0.795 (garage-inflated) statistical balanced 0.625, top-1 0.868 heuristic top-1 0.300 Only 489 train samples with severe class imbalance (Garage 565, Security 2, Weather 12). A blend sweep confirmed every non-zero CNN weight degrades the statistical model (0.625 -> 0.613 at 15% CNN, worse beyond), so the CNN is NOT wired into decode(). Production ensemble stays heuristic + statistical, both already live. Committing the trainer + shared pulse_encoder (trainer/inference parity) + metrics.json to document the reproducible negative result. The benched .pt/.onnx binaries are intentionally NOT committed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
36 lines
729 B
JSON
36 lines
729 B
JSON
{
|
|
"n_samples": 803,
|
|
"n_train": 489,
|
|
"n_val": 94,
|
|
"n_test": 220,
|
|
"classes": [
|
|
"Doorbell",
|
|
"Fan Controller",
|
|
"Garage Door Opener",
|
|
"Remote Control",
|
|
"Security Sensor",
|
|
"Weather Sensor"
|
|
],
|
|
"pulse_len": 512,
|
|
"epochs": 60,
|
|
"class_balance": {
|
|
"Doorbell": 39,
|
|
"Garage Door Opener": 565,
|
|
"Weather Sensor": 12,
|
|
"Fan Controller": 91,
|
|
"Security Sensor": 2,
|
|
"Remote Control": 94
|
|
},
|
|
"cnn": {
|
|
"accuracy": 0.7954545454545454,
|
|
"balanced_accuracy": 0.33796618290289177,
|
|
"macro_f1": 0.3552945963506287
|
|
},
|
|
"heuristic_same_test": {
|
|
"top1": 0.3
|
|
},
|
|
"statistical_same_test": {
|
|
"top1": 0.8681818181818182,
|
|
"balanced_accuracy": 0.6253728690437551
|
|
}
|
|
} |