leetcrypt
6945ece853
## Summary Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals. ## What Was Added ### Test Infrastructure - **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository - 3 weather sensors (LaCrosse, Acurite, Nexus) - 2 garage doors (LiftMaster 433MHz, Security+ 2.0) - 2 doorbells (GE, Byron) - 1 LED remote - 3 ceiling fan controls - **tests/real_captures/manifest.json** - Test case metadata with expected protocols - **scripts/benchmark_real.py** - Real-world benchmark runner - Loads manifest.json - Runs identification on each capture - Generates REAL_TEST_RESULTS.md - Compares with synthetic results ### Documentation - **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy) - **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis ## Results ### Performance Comparison | Metric | Synthetic | Real-World | Delta | |--------|-----------|------------|-------| | Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** | | Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** | | High Confidence | 66.7% | 9.1% | -57.6% | | No Matches | 8.3% | **45.5%** | +37.2% | ### Root Causes Identified 1. **Decoded File Format (45% of failures)** - CRITICAL - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW - Our system ONLY processes RAW_Data - All ceiling fan and LED remote files fail immediately - **Fix**: Add support for parsed decode .sub file format 2. **Missing Protocols (27% of failures)** - Acurite 02077M not in database (we have 609TXC) - GE Doorbell 19297 not in database - Byron DB421E not in database - LiftMaster not in database - **Fix**: Add these protocols from real captures 3. **Real Signal Complexity (27% of failures)** - LaCrosse real capture: 131 pulses (multi-packet) - Synthetic LaCrosse: 40 bits (single packet) - Real signals have noise, jitter, interference - **Fix**: Packet segmentation, higher noise tolerance 4. **Protocol Family Competition (9% of failures)** - Nexus found at rank 34 (beaten by Oregon Scientific) - Similar protocols competing instead of grouping - **Fix**: Protocol family scoring ## Key Insights ### Why Synthetic Worked (33% accuracy): - Perfect timing with controlled 5-15% jitter - Single packet per file - All RAW format - Known protocol parameters ### Why Real Failed (0% accuracy): - 45% already decoded (not RAW) - 27% missing from database - Variable signal quality - Multi-packet transmissions - Real-world interference **Conclusion**: System was optimized for unrealistic synthetic signals. ## Recommendations (Priority Order) 1. **P1 - Support Decoded Files** (+45% accuracy) - Parse Protocol, Bit, Key, TE fields - Match by protocol name + timing element - Skip RAW analysis for decoded files 2. **P2 - Add Missing Protocols** (+27% accuracy) - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X - Source from real captures or RTL_433 updates 3. **P3 - Improve Robustness** (+18% accuracy) - Packet segmentation for multi-transmission captures - Increase jitter tolerance to 25% - Repetition detection 4. **P4 - Family Grouping** (+9% accuracy) - Group similar protocols (Nexus/Oregon/Acurite) - Boost exact name matches **Expected Final Accuracy**: 60-70% after P1+P2 implementation. ## Files Changed - tests/real_captures/*.sub (11 files, 192KB) - tests/real_captures/manifest.json (11 test cases) - scripts/benchmark_real.py (370 lines) - REAL_TEST_RESULTS.md (benchmark results) - docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis) 🎯 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
11 lines
3.9 KiB
Plaintext
11 lines
3.9 KiB
Plaintext
Filetype: Flipper SubGhz RAW File
|
|
Version: 1
|
|
#
|
|
# Possible LiftMaster Skeleton Key?
|
|
#
|
|
Frequency: 433920000
|
|
Preset: FuriHalSubGhzPresetOok650Async
|
|
Protocol: RAW
|
|
RAW_Data: 461 -66 2011 -16946 99 -434 97 -166 99 -796 295 -134 425 -198 99 -596 1385 -166 40285 -66 3559 -66 5833 -66 2789 -66 82367 -66 161 -98 15489 -66 11071 -98 40405 -12004 95 -2834 99 -534 927 -168 133 -66 235 -264 1653 -66 4739 -66 5035 -66 1591 -66 3105 -66 11347 -132 9851 -17064 131 -1586 229 -1016 393 -66 327 -66 231 -100 131 -198 1883 -134 1525 -66 5475 -66 26571 -100 2365 -100 15441 -66 463 -134 3379 -100 3587 -68 1919 -100 4917 -66 5463 -64 10317 -134 24263 -18184 65 -660 195 -164 199 -328 197 -98 295 -66 131 -426 815 -98 197 -132 163 -326 3291 -166 1821 -66 327 -98 1641 -66 2897 -66 1787 -66 10149 -12310 97 -5000 65 -100 195 -1346 627 -134 67 -794 1489 -98 331 -17690 65 -596 163 -100 327 -132 297 -598 99 -134 497 -66 5795 -66 2083 -66 1647 -66 22611 -18340 65 -396 131 -398 431 -66 65 -298 133 -594 429 -66 497 -198 265 -136 1795 -66 20831 -100 2023 -100 13943 -66 3453 -16704 229 -166 131 -1302 789 -332 65 -530 889 -200 399 -66 1127 -100 165 -66 5629 -66 3687 -100 3567 -17796 131 -824 1091 -464 297 -66 1563 -164 3727 -16924 131 -658 199 -230 133 -200 231 -98 99 -732 397 -660 165 -66 331 -100 893 -330 397 -100 29791 -4292 1013 -1970 2001 -982 2955 -1016 2955 -982 2997 -988 2969 -1000 2989 -2946 1009 -1980 1991 -2970 977 -1012 2977 -35440 3023 -1964 1979 -1996 1975 -1000 2955 -2990 1009 -1956 1985 -2976 1011 -2966 1007 -1970 1997 -978 2977 -1988 1971 -38810 925 -2160 1843 -1070 2899 -1072 2933 -1006 2959 -1010 2963 -1006 2957 -2996 969 -1996 1971 -2990 977 -990 2981 -36818 2985 -2010 1959 -2010 1959 -984 2999 -2960 993 -2004 1965 -2966 1015 -2978 975 -1978 1989 -1000 2983 -1972 1977 -38752 957 -2056 1933 -1038 2937 -1042 2931 -1010 2965 -1008 2963 -1008 2959 -2960 1005 -1994 1971 -2982 975 -1024 2953 -36854 2993 -2006 1969 -1976 2001 -986 2979 -2968 999 -1968 2001 -2966 997 -2966 1007 -1990 1975 -1000 2981 -1970 1973 -38774 991 -2022 1943 -1004 2957 -1006 2993 -974 2995 -974 2991 -976 2965 -2990 1005 -1984 1977 -2978 975 -990 2987 -36782 3003 -1976 1999 -1978 1997 -982 2961 -2998 971 -2000 1975 -2988 975 -2974 1009 -1974 1991 -998 2953 -1988 1993 -151876 99 -594 131 -64 97 -430 131 -200 231 -534 133 -166 265 -198 3215 -64 1843 -66 5781 -66 19705 -66 10523 -98 1463 -100 16049 -98 25913 -100 3629 -98 1685 -66 4955 -68 10365 -17068 99 -528 131 -926 99 -658 425 -132 65 -392 231 -200 463 -166 65 -628 961 -66 4125 -66 5279 -132 1597 -66 13933 -66 34063 -100 425 -17448 163 -98 165 -392 99 -396 197 -134 201 -66 99 -132 295 -134 131 -166 67 -166 1227 -132 361 -100
|
|
RAW_Data: 21487 -14362 99 -3144 63 -66 263 -396 1679 -98 333 -66 233 -14838 265 -896 165 -100 131 -334 99 -132 1095 -68 463 -66 13871 -98 5189 -66 3761 -100 7075 -98 3363 -98 30143 -66 30455 -66 1521 -66 12419 -68 14061 -98 12577 -66 11139 -66 18717 -14890 67 -3036 197 -428 97 -624 657 -98 393 -330 65 -198 1285 -200 391 -68 4119 -66 4229 -15778 265 -66 265 -862 99 -170 893 -296 131 -198 133 -100 231 -132 6471 -66 4343 -17300 99 -268 131 -556 1187 -792 1379 -132 6083 -98 3443 -66 21627 -100 1227 -98 6901 -66 2171 -98 5167 -18424 165 -198 683 -1018 1277 -264 7675 -15910 97 -602 67 -23798 165 -134 65 -166 429 -232 297 -198 331 -164 527 -98 393 -100 463 -66 2149 -66 5645 -68 433 -100 3071 -98 3807 -66 4957 -68 12565 -68 4305 -17802 197 -98 163 -424 99 -164 229 -828 1093 -100 1463 -100 3851 -66 233 -66 1203 -66 3959 -66 3349 -100 15991 -66 1585 -66 5541 -98 3195 -66 10563 -98 1995 -100 24475 -68 1387 -100 229 -98 1887 -17628 231 -168 67 -594 397 -132 359 -98 357 -884 861 -98 231 -166 133 -134 3307 -66 1429 -66 2227 -64 4685 -98 15927 -66 9269 -66 14933 -18872 921 -296 65 -428 821 -98 99 -66 10077 -100 1925 -64 12683 -68 14321 -66 6695 -18744 199 -302 595 -100 197 -166 263 -98 533 -132 3517 -134 10241 -166 4981 -132 2007 -132 431 -66 10191 -98 5273 -66 1877 -100 10213 -68 19915 -100 44659
|