Initial commit: Phase 1 & Phase 2 infrastructure complete
This commit is contained in:
@@ -0,0 +1,569 @@
|
||||
# Signature Database Integration
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes how to integrate and use signature databases from Flipper Zero, RTL_433, and community sources for device identification.
|
||||
|
||||
## 1. Flipper Zero Sub-GHz Database
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/flipperdevices/flipperzero-firmware
|
||||
- **Location**: `/assets/resources/subghz/assets/` in firmware repo
|
||||
- **Format**: `.sub` files
|
||||
|
||||
### File Format Structure
|
||||
|
||||
#### Standard Key File
|
||||
```
|
||||
Filetype: Flipper SubGhz Key File
|
||||
Version: 1
|
||||
Frequency: 433920000
|
||||
Preset: FuriHalSubGhzPresetOok270Async
|
||||
Protocol: Princeton
|
||||
Bit: 24
|
||||
Key: 00 00 00 00 00 95 D5 D4
|
||||
TE: 400
|
||||
```
|
||||
|
||||
**Field Descriptions:**
|
||||
- `Filetype`: Must be "Flipper SubGhz Key File" for protocol files
|
||||
- `Version`: File format version (currently 1)
|
||||
- `Frequency`: Operating frequency in Hz (e.g., 433920000 = 433.92 MHz)
|
||||
- `Preset`: Modulation preset (see Preset Types below)
|
||||
- `Protocol`: Protocol name (Princeton, KeeLoq, Star Line, etc.)
|
||||
- `Bit`: Number of bits in the transmission
|
||||
- `Key`: Hex-encoded data payload
|
||||
- `TE`: Timing element in microseconds (pulse width)
|
||||
|
||||
#### RAW Signal File
|
||||
```
|
||||
Filetype: Flipper SubGhz RAW File
|
||||
Version: 1
|
||||
Frequency: 433920000
|
||||
Preset: FuriHalSubGhzPresetOok650Async
|
||||
Protocol: RAW
|
||||
RAW_Data: 29262 361 -68 2635 -66 24113 -66 11 -66 11 -132
|
||||
```
|
||||
|
||||
**RAW_Data Format:**
|
||||
- Array of timing values in microseconds
|
||||
- Positive values = carrier ON
|
||||
- Negative values = carrier OFF
|
||||
- Must alternate between positive and negative
|
||||
- Up to 512 values per line
|
||||
|
||||
#### BinRAW File (Compressed)
|
||||
```
|
||||
Filetype: Flipper SubGhz RAW File
|
||||
Version: 1
|
||||
Frequency: 315000000
|
||||
Preset: FuriHalSubGhzPreset2FSKDev238Async
|
||||
Protocol: BinRAW
|
||||
Bit: 1572
|
||||
TE: 597
|
||||
Bit_RAW: 260
|
||||
Data_RAW: 00 00 00 00 AA AA AA AA 0F 4A B5 55
|
||||
```
|
||||
|
||||
**BinRAW Format:**
|
||||
- `Bit`: Total bits in transmission
|
||||
- `TE`: Timing element (microsecond per bit)
|
||||
- `Bit_RAW`: Number of bits in compressed format
|
||||
- `Data_RAW`: Bit-packed data (1=carrier, 0=gap)
|
||||
|
||||
### Preset Types
|
||||
|
||||
| Preset | Modulation | Bandwidth | Deviation | Use Case |
|
||||
|--------|------------|-----------|-----------|----------|
|
||||
| `FuriHalSubGhzPresetOok270Async` | OOK | 270 kHz | - | Standard garage doors, remotes |
|
||||
| `FuriHalSubGhzPresetOok650Async` | OOK | 650 kHz | - | Fast protocols, doorbells |
|
||||
| `FuriHalSubGhzPreset2FSKDev238Async` | 2FSK | 270 kHz | 2.38 kHz | TPMS, some sensors |
|
||||
| `FuriHalSubGhzPreset2FSKDev476Async` | 2FSK | 270 kHz | 47.6 kHz | High-deviation FSK |
|
||||
| `FuriHalSubGhzPresetCustom` | Custom | Varies | Varies | User-defined configs |
|
||||
|
||||
### Custom Presets
|
||||
|
||||
Custom presets allow specific CC1101 register configurations:
|
||||
|
||||
```
|
||||
Filetype: Flipper SubGhz Key File
|
||||
Version: 1
|
||||
Frequency: 868350000
|
||||
Preset: FuriHalSubGhzPresetCustom
|
||||
Custom_preset_module: CC1101
|
||||
Custom_preset_data: 02 0D 03 07 08 32 0B 06 14 00 13 00 12 00 11 32 10 17 18 18 19 18 1D 91 1C 00 1B 07 20 FB 22 11 21 B6 00 00 C0 00 00 00 00 00 00 00
|
||||
Protocol: StarLine
|
||||
...
|
||||
```
|
||||
|
||||
### Common Protocols
|
||||
|
||||
| Protocol | Frequency | Modulation | Bit Length | Use Case |
|
||||
|----------|-----------|------------|------------|----------|
|
||||
| Princeton | 433.92 MHz | OOK_PWM | 24 | Generic remotes, garage doors |
|
||||
| KeeLoq | 433.92 MHz | OOK | 64-66 | Car key fobs, secure remotes |
|
||||
| Star Line | 433.92 MHz | OOK | Varies | Car alarm systems |
|
||||
| Came | 433.92 MHz | OOK | 12 | Gate openers |
|
||||
| Nice FLO | 433.92 MHz | OOK | 12-24 | Gate openers |
|
||||
| Somfy Telis | 433.42 MHz | OOK | 56 | Window blinds |
|
||||
| Holtek HT12X | 433.92 MHz | OOK_PWM | 12 | Generic remotes |
|
||||
|
||||
### Importing Flipper Signatures
|
||||
|
||||
#### Step 1: Clone the Repository
|
||||
```bash
|
||||
cd signatures/flipper
|
||||
git clone --depth 1 https://github.com/flipperdevices/flipperzero-firmware.git temp
|
||||
cp -r temp/assets/resources/subghz/assets/* ./
|
||||
rm -rf temp
|
||||
```
|
||||
|
||||
#### Step 2: Parse .sub Files
|
||||
```python
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def parse_flipper_sub(file_path):
|
||||
"""Parse a Flipper Zero .sub file"""
|
||||
data = {}
|
||||
with open(file_path, 'r') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if ':' in line:
|
||||
key, value = line.split(':', 1)
|
||||
data[key.strip()] = value.strip()
|
||||
return data
|
||||
|
||||
# Example usage
|
||||
sub_file = Path('signatures/flipper/princeton_433.sub')
|
||||
parsed = parse_flipper_sub(sub_file)
|
||||
print(f"Protocol: {parsed.get('Protocol')}")
|
||||
print(f"Frequency: {parsed.get('Frequency')} Hz")
|
||||
print(f"Key: {parsed.get('Key')}")
|
||||
```
|
||||
|
||||
#### Step 3: Import to Database
|
||||
```python
|
||||
def import_flipper_signature(parsed_data, device_id):
|
||||
"""Import parsed .sub file to database"""
|
||||
signature = {
|
||||
'device_id': device_id,
|
||||
'protocol': parsed_data.get('Protocol'),
|
||||
'frequency': int(parsed_data.get('Frequency', 0)),
|
||||
'modulation': parse_preset_modulation(parsed_data.get('Preset')),
|
||||
'bit_pattern': bytes.fromhex(parsed_data.get('Key', '').replace(' ', '')),
|
||||
'timing_min': int(parsed_data.get('TE', 0)) * 0.9, # 10% tolerance
|
||||
'timing_max': int(parsed_data.get('TE', 0)) * 1.1,
|
||||
'source': 'flipper',
|
||||
'source_file': str(file_path)
|
||||
}
|
||||
# Insert into database
|
||||
db.signatures.insert(signature)
|
||||
```
|
||||
|
||||
## 2. RTL_433 Protocol Database
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/merbanan/rtl_433
|
||||
- **Location**: `/src/devices/*.c` (protocol implementations)
|
||||
- **Documentation**: https://triq.org/rtl_433/
|
||||
|
||||
### Protocol Structure
|
||||
|
||||
RTL_433 protocols are defined in C code with decoder specifications:
|
||||
|
||||
```c
|
||||
static char* output_fields[] = {
|
||||
"model",
|
||||
"id",
|
||||
"channel",
|
||||
"battery_ok",
|
||||
"temperature_C",
|
||||
"humidity",
|
||||
"mic",
|
||||
NULL,
|
||||
};
|
||||
|
||||
r_device acurite_tower = {
|
||||
.name = "Acurite-Tower",
|
||||
.modulation = OOK_PULSE_PWM,
|
||||
.short_width = 220,
|
||||
.long_width = 440,
|
||||
.reset_limit = 900,
|
||||
.decode_fn = &acurite_tower_decode,
|
||||
.fields = output_fields,
|
||||
};
|
||||
```
|
||||
|
||||
### JSON Output Format
|
||||
|
||||
RTL_433 outputs decoded data as JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"time": "2025-01-11 20:15:32",
|
||||
"model": "Acurite-Tower",
|
||||
"id": 12345,
|
||||
"channel": "A",
|
||||
"battery_ok": 1,
|
||||
"temperature_C": 22.5,
|
||||
"humidity": 45,
|
||||
"mic": "CRC"
|
||||
}
|
||||
```
|
||||
|
||||
### Common Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `time` | String | Timestamp in ISO 8601 format |
|
||||
| `model` | String | Manufacturer-Model identifier |
|
||||
| `id` | Integer | Unique device ID |
|
||||
| `channel` | String | Channel identifier (A, B, C, etc.) |
|
||||
| `battery_ok` | Integer | Battery status (0=low, 1=ok) |
|
||||
| `temperature_C` | Float | Temperature in Celsius |
|
||||
| `humidity` | Integer | Relative humidity percentage |
|
||||
| `mic` | String | Message integrity check type |
|
||||
|
||||
### Modulation Types
|
||||
|
||||
| Modulation | Description | Common Devices |
|
||||
|------------|-------------|----------------|
|
||||
| `OOK_PWM` | Pulse Width Modulation | Remotes, sensors |
|
||||
| `OOK_PPM` | Pulse Position Modulation | Weather stations |
|
||||
| `OOK_PCM` | Pulse Code Modulation | Security sensors |
|
||||
| `FSK_PCM` | FSK Pulse Code | TPMS, smart meters |
|
||||
| `FSK_PWM` | FSK Pulse Width | Advanced sensors |
|
||||
|
||||
### Extracting Protocol Definitions
|
||||
|
||||
#### Step 1: Extract from Source Code
|
||||
```bash
|
||||
cd signatures/rtl433
|
||||
git clone --depth 1 https://github.com/merbanan/rtl_433.git temp
|
||||
grep -r "r_device" temp/src/devices/*.c > protocols.txt
|
||||
rm -rf temp
|
||||
```
|
||||
|
||||
#### Step 2: Parse Protocol Definitions
|
||||
```python
|
||||
import re
|
||||
import json
|
||||
|
||||
def parse_rtl433_protocol(protocol_string):
|
||||
"""Extract protocol definition from C code"""
|
||||
pattern = r'r_device\s+(\w+)\s*=\s*{([^}]+)}'
|
||||
match = re.search(pattern, protocol_string, re.DOTALL)
|
||||
|
||||
if not match:
|
||||
return None
|
||||
|
||||
name = match.group(1)
|
||||
body = match.group(2)
|
||||
|
||||
# Extract fields
|
||||
modulation = re.search(r'\.modulation\s*=\s*(\w+)', body)
|
||||
short_width = re.search(r'\.short_width\s*=\s*(\d+)', body)
|
||||
long_width = re.search(r'\.long_width\s*=\s*(\d+)', body)
|
||||
model_name = re.search(r'\.name\s*=\s*"([^"]+)"', body)
|
||||
|
||||
return {
|
||||
'protocol_name': name,
|
||||
'model': model_name.group(1) if model_name else None,
|
||||
'modulation': modulation.group(1) if modulation else None,
|
||||
'short_width': int(short_width.group(1)) if short_width else None,
|
||||
'long_width': int(long_width.group(1)) if long_width else None
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Generate Protocol Database
|
||||
```python
|
||||
# Read all protocol definitions
|
||||
protocols = []
|
||||
for c_file in Path('temp/src/devices').glob('*.c'):
|
||||
with open(c_file, 'r') as f:
|
||||
content = f.read()
|
||||
parsed = parse_rtl433_protocol(content)
|
||||
if parsed:
|
||||
protocols.append(parsed)
|
||||
|
||||
# Save as JSON
|
||||
with open('signatures/rtl433/protocols.json', 'w') as f:
|
||||
json.dump(protocols, f, indent=2)
|
||||
```
|
||||
|
||||
### Using rtl_433 Test Data
|
||||
|
||||
RTL_433 provides test signals with expected JSON output:
|
||||
|
||||
```bash
|
||||
# Clone test repository
|
||||
git clone https://github.com/merbanan/rtl_433_tests.git signatures/rtl433/tests
|
||||
|
||||
# Test files are organized as:
|
||||
# rtl_433_tests/tests/{protocol_name}/{signal_type}/*.cu8
|
||||
# rtl_433_tests/tests/{protocol_name}/{signal_type}/*.json
|
||||
```
|
||||
|
||||
Example test data structure:
|
||||
```
|
||||
rtl_433_tests/tests/acurite/01/
|
||||
├── g001_433.92M_250k.cu8 # Raw signal capture
|
||||
└── g001_433.92M_250k.json # Expected decoded output
|
||||
```
|
||||
|
||||
### Importing RTL_433 Signatures
|
||||
|
||||
```python
|
||||
def import_rtl433_protocol(protocol_data, device_id):
|
||||
"""Import RTL_433 protocol to database"""
|
||||
# Determine frequency from modulation
|
||||
frequency_map = {
|
||||
'OOK': 433920000,
|
||||
'FSK': 868000000 # Varies by device
|
||||
}
|
||||
|
||||
modulation_type = protocol_data.get('modulation', 'OOK')
|
||||
base_freq = frequency_map.get(modulation_type[:3], 433920000)
|
||||
|
||||
signature = {
|
||||
'device_id': device_id,
|
||||
'protocol': protocol_data.get('model'),
|
||||
'frequency': base_freq,
|
||||
'modulation': modulation_type,
|
||||
'timing_min': protocol_data.get('short_width'),
|
||||
'timing_max': protocol_data.get('long_width'),
|
||||
'source': 'rtl433',
|
||||
'source_file': protocol_data.get('source_file')
|
||||
}
|
||||
|
||||
# Insert into rtl433_protocols table
|
||||
db.rtl433_protocols.insert({
|
||||
'protocol_name': protocol_data.get('protocol_name'),
|
||||
'model': protocol_data.get('model'),
|
||||
'modulation': modulation_type,
|
||||
'short_width': protocol_data.get('short_width'),
|
||||
'long_width': protocol_data.get('long_width'),
|
||||
'json_fields': protocol_data.get('fields', {})
|
||||
})
|
||||
|
||||
# Also create general signature
|
||||
db.signatures.insert(signature)
|
||||
```
|
||||
|
||||
## 3. Universal Radio Hacker (URH)
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/jopohl/urh
|
||||
- **Format**: `.urh` project files, `.complex` signal files
|
||||
|
||||
### Signal File Formats
|
||||
|
||||
#### .complex Files
|
||||
Raw I/Q signal data in binary format:
|
||||
- Interleaved I and Q samples
|
||||
- Typically 32-bit float or 8-bit integer
|
||||
- Sample rate metadata in accompanying .cfl file
|
||||
|
||||
#### .urh Project Files
|
||||
XML-based project containing:
|
||||
- Signal definitions
|
||||
- Demodulation parameters
|
||||
- Protocol structure
|
||||
- Labeled message fields
|
||||
|
||||
### Extracting URH Protocols
|
||||
|
||||
URH projects often contain valuable protocol information in the community wiki.
|
||||
|
||||
```bash
|
||||
# Download community-shared URH projects
|
||||
cd signatures/urh
|
||||
# Check URH GitHub wiki for shared project links
|
||||
```
|
||||
|
||||
## 4. Community Signatures
|
||||
|
||||
### User Submission Format
|
||||
|
||||
When users identify a device, they can submit:
|
||||
|
||||
```json
|
||||
{
|
||||
"device": {
|
||||
"manufacturer": "Chamberlain",
|
||||
"model": "KLIK3U-SS",
|
||||
"type": "garage_door_opener",
|
||||
"fcc_id": "K49KLIK3U"
|
||||
},
|
||||
"signature": {
|
||||
"frequency": 433920000,
|
||||
"protocol": "Security+ 2.0",
|
||||
"modulation": "OOK",
|
||||
"notes": "Rolling code, 3-button remote"
|
||||
},
|
||||
"evidence": {
|
||||
"photos": ["device_front.jpg", "device_back.jpg", "fcc_label.jpg"],
|
||||
"capture_file": "chamberlain_klik3u_001.sub"
|
||||
},
|
||||
"location": {
|
||||
"latitude": 40.7128,
|
||||
"longitude": -74.0060,
|
||||
"accuracy": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Verification Process
|
||||
|
||||
1. User submits identification with photo evidence
|
||||
2. Community members vote (upvote/downvote)
|
||||
3. After 5 net upvotes, identification is auto-verified
|
||||
4. Verified identifications create new signature entries
|
||||
5. High-reputation users can verify immediately
|
||||
|
||||
## 5. Signature Matching Algorithm
|
||||
|
||||
### Matching Strategy
|
||||
|
||||
```python
|
||||
def match_capture_to_signatures(capture):
|
||||
"""
|
||||
Match a capture against known signatures
|
||||
Returns list of (device_id, confidence) tuples
|
||||
"""
|
||||
matches = []
|
||||
|
||||
# Exact protocol + frequency + timing match
|
||||
exact = db.query('''
|
||||
SELECT device_id, 1.0 as confidence
|
||||
FROM signatures
|
||||
WHERE protocol = ? AND frequency = ? AND ? BETWEEN timing_min AND timing_max
|
||||
''', [capture.protocol, capture.frequency, capture.timing_element])
|
||||
matches.extend(exact)
|
||||
|
||||
# Protocol + frequency match (80% confidence)
|
||||
protocol_freq = db.query('''
|
||||
SELECT device_id, 0.8 as confidence
|
||||
FROM signatures
|
||||
WHERE protocol = ? AND frequency = ?
|
||||
''', [capture.protocol, capture.frequency])
|
||||
matches.extend(protocol_freq)
|
||||
|
||||
# Bit pattern matching (if available)
|
||||
if capture.key_data:
|
||||
pattern_matches = match_bit_pattern(capture.key_data)
|
||||
matches.extend(pattern_matches)
|
||||
|
||||
# De-duplicate and sort by confidence
|
||||
unique_matches = {}
|
||||
for device_id, conf in matches:
|
||||
if device_id not in unique_matches or conf > unique_matches[device_id]:
|
||||
unique_matches[device_id] = conf
|
||||
|
||||
return sorted(unique_matches.items(), key=lambda x: x[1], reverse=True)
|
||||
```
|
||||
|
||||
### Bit Pattern Matching
|
||||
|
||||
```python
|
||||
def match_bit_pattern(key_data, signatures):
|
||||
"""
|
||||
Match key data against signature patterns with masks
|
||||
"""
|
||||
matches = []
|
||||
|
||||
for sig in signatures:
|
||||
if not sig.bit_mask:
|
||||
continue
|
||||
|
||||
# Apply mask and compare
|
||||
masked_capture = apply_mask(key_data, sig.bit_mask)
|
||||
masked_signature = apply_mask(sig.bit_pattern, sig.bit_mask)
|
||||
|
||||
if masked_capture == masked_signature:
|
||||
confidence = 0.9 * sig.weight
|
||||
matches.append((sig.device_id, confidence))
|
||||
|
||||
return matches
|
||||
|
||||
def apply_mask(data, mask):
|
||||
"""Bitwise AND operation on byte arrays"""
|
||||
return bytes(a & b for a, b in zip(data, mask))
|
||||
```
|
||||
|
||||
## 6. Database Import Scripts
|
||||
|
||||
### Complete Import Pipeline
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Import all signature databases"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from importers import flipper, rtl433, urh
|
||||
|
||||
def main():
|
||||
print("Starting signature database import...")
|
||||
|
||||
# Import Flipper Zero signatures
|
||||
print("\n[1/3] Importing Flipper Zero .sub files...")
|
||||
flipper_dir = Path('signatures/flipper')
|
||||
flipper_count = flipper.import_all(flipper_dir)
|
||||
print(f" Imported {flipper_count} Flipper signatures")
|
||||
|
||||
# Import RTL_433 protocols
|
||||
print("\n[2/3] Importing RTL_433 protocols...")
|
||||
rtl433_file = Path('signatures/rtl433/protocols.json')
|
||||
rtl433_count = rtl433.import_protocols(rtl433_file)
|
||||
print(f" Imported {rtl433_count} RTL_433 protocols")
|
||||
|
||||
# Import URH community signals
|
||||
print("\n[3/3] Importing URH signals...")
|
||||
urh_dir = Path('signatures/urh')
|
||||
urh_count = urh.import_all(urh_dir)
|
||||
print(f" Imported {urh_count} URH signatures")
|
||||
|
||||
print(f"\nTotal signatures imported: {flipper_count + rtl433_count + urh_count}")
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
```
|
||||
|
||||
## 7. Signature Database Maintenance
|
||||
|
||||
### Regular Updates
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/update_signatures.sh
|
||||
|
||||
cd signatures/flipper
|
||||
git pull
|
||||
|
||||
cd ../rtl433
|
||||
git pull
|
||||
|
||||
# Re-import updated signatures
|
||||
python3 scripts/import_signatures.py --update
|
||||
```
|
||||
|
||||
### Quality Metrics
|
||||
|
||||
Track signature effectiveness:
|
||||
|
||||
```sql
|
||||
-- Signature match success rate
|
||||
SELECT
|
||||
s.id,
|
||||
d.manufacturer,
|
||||
d.model,
|
||||
COUNT(c.id) as total_matches,
|
||||
AVG(c.match_confidence) as avg_confidence
|
||||
FROM signatures s
|
||||
JOIN devices d ON s.device_id = d.id
|
||||
LEFT JOIN captures c ON c.device_id = d.id AND c.match_method = 'auto'
|
||||
GROUP BY s.id, d.manufacturer, d.model
|
||||
ORDER BY total_matches DESC;
|
||||
```
|
||||
Reference in New Issue
Block a user