Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement pattern-based decoder for single-transmission RF captures

Browse Source 
Added pattern-based decoding system specifically designed for short captures
from Flipper Zero and LilyGo T-Embed devices that don't have enough repetitions
for RTL_433.

## New Components:

1. **Protocol Database** (protocol_database.py)
   - 18 known RF protocol signatures
   - Categories: Weather Sensors, Garage Doors, TPMS, Doorbells, etc.
   - Timing patterns for Acurite, Oregon Scientific, LaCrosse, Nexus, etc.

2. **Pattern Decoder** (pattern_decoder.py)
   - Multi-strategy decoder using 3 approaches:
     - Timing pattern analysis (K-means clustering for SHORT/LONG pulses)
     - Statistical fingerprinting (signal characteristics)
     - Protocol library matching
   - Works with single-transmission captures (100-500 pulses)

3. **Matcher Integration** (strategies.py)
   - Added PatternBasedStrategy to matcher pipeline
   - Integrates with existing MatchResult system
   - Confidence scoring: 0.5-0.9 based on match quality

## Test Results:

**Pattern Decoder vs RTL_433 Performance:**
- RTL_433: 0% decode rate (0/8 files) - requires multiple repetitions
- Pattern Decoder: 44.4% decode rate (4/9 files) - works with single captures

**Successful Decodes:**
- Oregon Scientific weather sensors (76% confidence)
- Acurite weather stations (52% confidence)
- 24 total device matches across 4 files

## Implementation Details:

- K-means clustering for pulse width identification
- Statistical fingerprinting with mean, std, duty cycle
- Protocol database with 7 weather sensors + 11 other device types
- Confidence thresholds optimized for single-tx captures
- Fallback to sklearn K-means or percentile-based clustering

## Documentation:

- PATTERN_BASED_DECODING_PLAN.md: Complete implementation plan
- test_pattern_decoder.py: Comprehensive test suite

Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
priestlypython
2026-01-14 19:22:07 -08:00
parent 8092d59b8d
commit 9be14af160
5 changed files with 2004 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/matcher/pattern_decoder.py
+461
View File
@@ -0,0 +1,461 @@
#!/usr/bin/env python3
"""
Pattern-Based Decoder for RF Signals
Decodes devices from single-transmission captures using timing patterns,
statistical fingerprints, and protocol library matching.
Designed specifically for short captures from Flipper Zero and LilyGo T-Embed
that don't have enough repetitions for RTL_433.
"""
import numpy as np
from dataclasses import dataclass
from typing import List, Optional, Tuple, Dict
from collections import Counter
from src.parser.sub_parser import SignalMetadata
from src.matcher.protocol_database import (
ProtocolSignature,
ProtocolDatabase,
get_protocol_database
)
@dataclass
class PulseStatistics:
"""Statistical characteristics of pulse data"""
mean_pulse_width: float
std_pulse_width: float
mean_gap_width: float
std_gap_width: float
pulse_gap_ratio: float
duty_cycle: float
pulse_count: int
min_pulse: int
max_pulse: int
min_gap: int
max_gap: int
@dataclass
class TimingPattern:
"""Detected timing pattern in signal"""
short_pulse_us: int
long_pulse_us: int
short_gap_us: int
long_gap_us: int
bit_pattern: str # Binary string decoded from pulses
confidence: float
@dataclass
class DeviceMatch:
"""Potential device match from pattern decoder"""
protocol: ProtocolSignature
confidence: float
match_method: str # 'timing', 'fingerprint', 'protocol'
details: Dict
@property
def name(self) -> str:
return self.protocol.name
@property
def manufacturer(self) -> Optional[str]:
return self.protocol.manufacturer
@property
def category(self) -> str:
return self.protocol.category
class PatternDecoder:
"""
Multi-strategy decoder for single-transmission RF captures
Uses three complementary strategies:
1. Timing Pattern Analysis - Identify SHORT/LONG pulses and decode bits
2. Statistical Fingerprinting - Match signal characteristics
3. Protocol Library Matching - Compare against known protocol signatures
"""
def __init__(self, protocol_db: Optional[ProtocolDatabase] = None):
self.protocol_db = protocol_db or get_protocol_database()
def decode(self, metadata: SignalMetadata) -> List[DeviceMatch]:
"""
Decode device from signal metadata using multi-strategy approach
Args:
metadata: Parsed .sub file metadata with RAW_Data
Returns:
List of DeviceMatch sorted by confidence (highest first)
"""
if not metadata.has_raw_data:
return []
pulses = metadata.raw_data
frequency = metadata.frequency
matches = []
# Strategy 1: Timing Pattern Analysis
timing_matches = self._decode_timing_patterns(pulses, frequency)
matches.extend(timing_matches)
# Strategy 2: Statistical Fingerprinting
fingerprint = self._extract_fingerprint(pulses)
fingerprint_matches = self._match_fingerprint(fingerprint, frequency)
matches.extend(fingerprint_matches)
# Strategy 3: Protocol Library Matching (combines timing + known protocols)
# Already integrated into timing_matches above
# Deduplicate and rank by confidence
return self._rank_matches(matches)
def _identify_pulse_widths(self, pulses: List[int]) -> Tuple[int, int, int, int]:
"""
Identify SHORT/LONG pulse and gap durations using K-means clustering
Returns:
(short_pulse, long_pulse, short_gap, long_gap) in microseconds
"""
if not pulses:
return (0, 0, 0, 0)
# Separate positive (HIGH pulses) and negative (LOW gaps)
high_pulses = [abs(p) for p in pulses if p > 0]
low_pulses = [abs(p) for p in pulses if p < 0]
# Cluster high pulses into SHORT/LONG
short_pulse, long_pulse = self._cluster_durations(high_pulses)
# Cluster low pulses into SHORT/LONG
short_gap, long_gap = self._cluster_durations(low_pulses)
return (short_pulse, long_pulse, short_gap, long_gap)
def _cluster_durations(self, durations: List[int]) -> Tuple[int, int]:
"""
Cluster durations into two groups (SHORT and LONG) using K-means
Falls back to percentile-based approach if K-means unavailable
"""
if not durations:
return (0, 0)
if len(durations) < 2:
val = durations[0]
return (val, val)
# Try K-means clustering (sklearn)
try:
from sklearn.cluster import KMeans
# Convert to 2D array for sklearn
X = np.array(durations).reshape(-1, 1)
# Cluster into 2 groups
kmeans = KMeans(n_clusters=2, random_state=42, n_init=10)
kmeans.fit(X)
# Get cluster centers
centers = sorted(kmeans.cluster_centers_.flatten())
return (int(centers[0]), int(centers[1]))
except ImportError:
# Fallback: Use percentile-based approach
sorted_durations = sorted(durations)
median_idx = len(sorted_durations) // 2
# Split at median
lower_half = sorted_durations[:median_idx]
upper_half = sorted_durations[median_idx:]
short = int(np.mean(lower_half)) if lower_half else sorted_durations[0]
long = int(np.mean(upper_half)) if upper_half else sorted_durations[-1]
return (short, long)
def _decode_timing_patterns(
self,
pulses: List[int],
frequency: int
) -> List[DeviceMatch]:
"""
Decode signal using timing pattern analysis
Steps:
1. Identify SHORT/LONG pulse widths
2. Decode binary pattern (SHORT=0, LONG=1)
3. Match against protocol database
"""
matches = []
# Identify pulse widths
short_pulse, long_pulse, short_gap, long_gap = self._identify_pulse_widths(pulses)
if short_pulse == 0 or long_pulse == 0:
return []
# Decode to binary pattern (using HIGH pulses only)
bit_pattern = self._decode_to_bits(pulses, short_pulse, long_pulse)
# Find protocols matching these timings
protocol_matches = self.protocol_db.find_by_timing(
short_pulse,
long_pulse,
frequency
)
for proto in protocol_matches:
# Calculate confidence based on:
# 1. Timing accuracy
# 2. Bit count match
# 3. Pattern match (if preamble/sync defined)
timing_error = abs(proto.short_pulse_us - short_pulse) / proto.short_pulse_us
timing_confidence = max(0, 1.0 - timing_error)
bit_count = len(bit_pattern)
bit_count_match = (proto.min_bits <= bit_count <= proto.max_bits)
bit_confidence = 1.0 if bit_count_match else 0.5
# Check preamble/sync patterns
pattern_confidence = 1.0
if proto.preamble_pattern and proto.preamble_pattern in bit_pattern:
pattern_confidence = 1.0
elif proto.sync_pattern and proto.sync_pattern in bit_pattern:
pattern_confidence = 0.9
else:
pattern_confidence = 0.7
# Overall confidence (weighted average)
overall_confidence = (
timing_confidence * 0.4 +
bit_confidence * 0.3 +
pattern_confidence * 0.3
)
if overall_confidence >= proto.min_confidence:
matches.append(DeviceMatch(
protocol=proto,
confidence=overall_confidence,
match_method='timing_pattern',
details={
'short_pulse_us': short_pulse,
'long_pulse_us': long_pulse,
'bit_count': bit_count,
'bit_pattern': bit_pattern[:64], # Truncate for readability
'timing_error': f"{timing_error:.2%}",
}
))
return matches
def _decode_to_bits(
self,
pulses: List[int],
short_pulse: int,
long_pulse: int
) -> str:
"""
Decode pulse train to binary string
Uses PWM encoding: SHORT=0, LONG=1
"""
bits = []
threshold = (short_pulse + long_pulse) / 2
for pulse in pulses:
if pulse > 0: # Only decode HIGH pulses
duration = abs(pulse)
if duration < threshold:
bits.append('0')
else:
bits.append('1')
return ''.join(bits)
def _extract_fingerprint(self, pulses: List[int]) -> PulseStatistics:
"""
Extract statistical fingerprint from pulse data
Returns metrics that characterize the signal's timing properties
"""
if not pulses:
return PulseStatistics(
mean_pulse_width=0, std_pulse_width=0,
mean_gap_width=0, std_gap_width=0,
pulse_gap_ratio=0, duty_cycle=0,
pulse_count=0,
min_pulse=0, max_pulse=0,
min_gap=0, max_gap=0
)
# Separate HIGH pulses and LOW gaps
high_pulses = [p for p in pulses if p > 0]
low_pulses = [abs(p) for p in pulses if p < 0]
# Calculate statistics
mean_pulse = np.mean(high_pulses) if high_pulses else 0
std_pulse = np.std(high_pulses) if high_pulses else 0
mean_gap = np.mean(low_pulses) if low_pulses else 0
std_gap = np.std(low_pulses) if low_pulses else 0
total_high = sum(high_pulses)
total_low = sum(low_pulses)
total_time = total_high + total_low
pulse_gap_ratio = mean_pulse / mean_gap if mean_gap > 0 else 0
duty_cycle = total_high / total_time if total_time > 0 else 0
return PulseStatistics(
mean_pulse_width=float(mean_pulse),
std_pulse_width=float(std_pulse),
mean_gap_width=float(mean_gap),
std_gap_width=float(std_gap),
pulse_gap_ratio=float(pulse_gap_ratio),
duty_cycle=float(duty_cycle),
pulse_count=len(pulses),
min_pulse=min(high_pulses) if high_pulses else 0,
max_pulse=max(high_pulses) if high_pulses else 0,
min_gap=min(low_pulses) if low_pulses else 0,
max_gap=max(low_pulses) if low_pulses else 0,
)
def _match_fingerprint(
self,
fingerprint: PulseStatistics,
frequency: int
) -> List[DeviceMatch]:
"""
Match statistical fingerprint against protocol database
Uses signal characteristics to find similar protocols
"""
matches = []
# Get protocols near this frequency
candidates = self.protocol_db.find_by_frequency(frequency)
for proto in candidates:
# Compare fingerprint characteristics
# Use typical timing to estimate expected characteristics
# Expected mean pulse ~ (short + long) / 2
expected_mean_pulse = (proto.short_pulse_us + proto.long_pulse_us) / 2
pulse_error = abs(expected_mean_pulse - fingerprint.mean_pulse_width) / expected_mean_pulse
# Expected pulse count
expected_count = proto.typical_pulse_count
count_error = abs(expected_count - fingerprint.pulse_count) / expected_count
# Similarity score (lower error = higher confidence)
pulse_similarity = max(0, 1.0 - pulse_error)
count_similarity = max(0, 1.0 - count_error)
overall_confidence = (pulse_similarity * 0.6 + count_similarity * 0.4)
# Lower threshold for fingerprint matches (less precise)
if overall_confidence >= 0.4:
matches.append(DeviceMatch(
protocol=proto,
confidence=overall_confidence * 0.8, # Scale down (less confident)
match_method='fingerprint',
details={
'mean_pulse_us': f"{fingerprint.mean_pulse_width:.1f}",
'pulse_count': fingerprint.pulse_count,
'duty_cycle': f"{fingerprint.duty_cycle:.2%}",
'pulse_error': f"{pulse_error:.2%}",
'count_error': f"{count_error:.2%}",
}
))
return matches
def _rank_matches(self, matches: List[DeviceMatch]) -> List[DeviceMatch]:
"""
Deduplicate and rank matches by confidence
If same protocol matched by multiple methods, keep highest confidence
"""
# Group by protocol name
by_protocol: Dict[str, List[DeviceMatch]] = {}
for match in matches:
name = match.protocol.name
if name not in by_protocol:
by_protocol[name] = []
by_protocol[name].append(match)
# Keep best match per protocol
best_matches = []
for name, protocol_matches in by_protocol.items():
best = max(protocol_matches, key=lambda m: m.confidence)
best_matches.append(best)
# Sort by confidence (highest first)
return sorted(best_matches, key=lambda m: m.confidence, reverse=True)
def get_version(self) -> str:
"""Get decoder version"""
return "PatternDecoder v1.0"
# Global instance
_decoder: Optional[PatternDecoder] = None
def get_pattern_decoder() -> PatternDecoder:
"""Get singleton pattern decoder instance"""
global _decoder
if _decoder is None:
_decoder = PatternDecoder()
return _decoder
if __name__ == '__main__':
# Test pattern decoder with sample data
from src.parser.sub_parser import parse_sub_file
from pathlib import Path
print("=== Pattern Decoder Test ===")
print()
# Try to load a test file
test_file = Path("data/test_rtl433_real/LaCrosse-TX141TH-BV2-raw.sub")
if test_file.exists():
print(f"Testing with: {test_file.name}")
print()
metadata = parse_sub_file(str(test_file))
decoder = get_pattern_decoder()
print(f"Signal Details:")
print(f" Frequency: {metadata.frequency / 1_000_000:.3f} MHz")
print(f" Pulse Count: {metadata.pulse_count}")
print()
print("Running pattern decoder...")
matches = decoder.decode(metadata)
print(f"Found {len(matches)} potential matches:")
print()
for i, match in enumerate(matches[:5], 1): # Show top 5
print(f"Match #{i}:")
print(f" Device: {match.name}")
print(f" Manufacturer: {match.manufacturer or 'Unknown'}")
print(f" Category: {match.category}")
print(f" Confidence: {match.confidence:.2%}")
print(f" Method: {match.match_method}")
print(f" Details: {match.details}")
print()
else:
print(f"❌ Test file not found: {test_file}")
print(" Run download script first to get test data")