Trains a PyTorch 1D CNN on encoded RAW pulse arrays as the Phase 3B leg
of the designed ensemble, scored on the SAME group-disjoint held-out test
set as the heuristic and statistical models (GroupShuffleSplit by device
sub-folder, no near-duplicate leakage).
RESULT — CNN is data-starved and loses decisively:
CNN balanced 0.338, top-1 0.795 (garage-inflated)
statistical balanced 0.625, top-1 0.868
heuristic top-1 0.300
Only 489 train samples with severe class imbalance (Garage 565, Security 2,
Weather 12). A blend sweep confirmed every non-zero CNN weight degrades the
statistical model (0.625 -> 0.613 at 15% CNN, worse beyond), so the CNN is
NOT wired into decode(). Production ensemble stays heuristic + statistical,
both already live.
Committing the trainer + shared pulse_encoder (trainer/inference parity) +
metrics.json to document the reproducible negative result. The benched
.pt/.onnx binaries are intentionally NOT committed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trains sklearn RandomForest + GradientBoosting on 14 timing/statistical
features from RAW-only UberGuidoZ captures, with a group-aware split
(GroupShuffleSplit by device sub-folder) so near-duplicate captures never
leak across train/test. The heuristic CategoryRouter is scored on the exact
same held-out files for a fair comparison.
Best model (gradient_boost): 86.8% top-1 accuracy, 62.5% balanced accuracy
vs heuristic 30.0% top-1 / 55.9% routed. Raw accuracy is inflated by the
70%-Garage RAW class balance; balanced accuracy (~2x the heuristic) is the
honest number and meets the Phase 3A 60-75% target. Not yet wired into the
decoder ensemble.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt