giglez/docs/RF_TEST_DATABASES.md

# RF Signal Test Databases - Comprehensive Guide

## Summary

This document catalogs labeled RF signal databases found via Google dorking and GitHub searches for testing our RTL_433 integration implementation.

**Total Databases Found:** 6 primary + 10+ secondary sources
**Total Estimated Files:** 30,000+ .sub files, 1,000+ .cu8 files
**Best for Testing:** rtl_433_tests (100% labeled, known protocols)

---

## Primary Databases (Recommended)

### 1. RTL_433 Official Test Files ⭐⭐⭐⭐⭐

**Repository:** https://github.com/merbanan/rtl_433_tests
**Quality:** Highest (100% labeled, protocol-verified)
**Format:** `.cu8`, `.json`
**Size:** ~1,000 test files
**Coverage:** 244 protocols

#### Features:
- Official regression test suite for rtl_433
- Each file has corresponding `.json` with expected decode results
- Organized by manufacturer/device type
- Guaranteed to decode with RTL_433

#### Directory Structure:
```
tests/
├── acurite/
│   ├── 01/        # Acurite 606TX
│   ├── 02/        # Acurite Tower
│   └── ...
├── oregon_scientific/
│   ├── 01/        # Oregon v2.1
│   ├── 02/        # Oregon v3
│   └── ...
├── lacrosse/
├── tpms/
│   ├── ford/
│   ├── toyota/
│   └── ...
└── ...
```

#### Protocols Included:
- Weather Sensors: Acurite, Oregon, LaCrosse, Nexus
- TPMS: Ford, Toyota, GM, Schrader, Continental
- Doorbells: Byron, Honeywell
- Security: Visonic, DSC, Paradox
- And 200+ more

#### Testing Priority: **Highest**

**Problem:** Files are in `.cu8` format (complex I/Q samples), not `.sub` format
**Solution:** Need to create converter (see section below)

---

### 2. FlipperZero-Subghz-DB ⭐⭐⭐☆☆

**Repository:** https://github.com/Zero-Sploit/FlipperZero-Subghz-DB
**Quality:** Medium (mixed protocols, community-contributed)
**Format:** `.sub` (Flipper format - ready to use!)
**Size:** 13,717 .sub files, 721 folders
**Coverage:** Mostly garage/gate openers, some sensors

#### Features:
- Largest single collection of .sub files
- Organized by device type
- Ready to use (no conversion needed)
- Community-maintained

#### Directory Structure:
```
Sub-GHz/
├── Garages/
├── Gates/
├── Doorbells/
├── Car_Keys/
├── Weather_Stations/  ← Useful for our testing
├── TPMS/              ← Useful for our testing
└── ...
```

#### Testing Priority: **Medium**

**Pros:** Ready-to-use .sub format
**Cons:** Mostly proprietary protocols (low RTL_433 match rate expected)

---

### 3. UberGuidoZ Flipper Database ⭐⭐⭐⭐☆

**Repository:** https://github.com/UberGuidoZ/Flipper
**Quality:** High (well-organized, actively maintained)
**Format:** `.sub`, `.ir`, `.nfc`, `.bin`
**Size:** ~500MB (10,000+ files)
**Coverage:** Comprehensive multi-protocol collection

#### Features:
- Most comprehensive Flipper Zero resource
- Frequent updates
- Well-documented
- Includes sub-directories for specific devices

#### Notable Subdirectories:
```
Flipper/
├── Sub-GHz/
│   ├── Weather_Sensors/
│   ├── Remotes/
│   ├── Vehicles/
│   └── ...
├── Infrared/
├── NFC/
└── BadUSB/
```

#### Testing Priority: **High**

**Best Use:** Search for specific manufacturer names (Acurite, Oregon, LaCrosse)

---

### 4. Full Flipper Database ⭐⭐⭐☆☆

**Repository:** https://github.com/DefinetlyNotAI/Full_Flipper_Database
**Quality:** Medium (comprehensive but mixed quality)
**Format:** `.sub`, `.ir`, `.nfc`, etc
**Size:** Large (exact count unknown)
**Coverage:** "The one and only flipper zero Database for all your pen-testing needs"

#### Features:
- Comprehensive collection
- Regular updates
- Mixed quality (community contributions)

#### Testing Priority: **Medium**

**Best Use:** Backup source if other databases don't have specific device

---

## Secondary Databases (Specialized)

### 5. WSSFlipperZero (Weather Station Spoofer) ⭐⭐⭐⭐☆

**Repository:** https://github.com/timdigga/WSSFlipperZero
**Type:** Synthetic .sub generator
**Format:** Generated `.sub` files
**Protocols:** Acurite, LaCrosse, Oregon Scientific

#### Features:
- Generates valid weather sensor .sub files
- Uses OpenWeatherMap API for realistic data
- Known-good protocol implementations

#### Testing Priority: **High** (for synthetic testing)

**Use Case:** Generate test files with known parameters for validation

---

### 6. OWMFlipperZero (OpenWeatherMap Sync) ⭐⭐⭐☆☆

**Repository:** https://github.com/timdigga/OWMFlipperZero
**Type:** Weather .sub generator
**Format:** Generated `.sub` files

Similar to WSSFlipperZero, generates synthetic weather sensor signals.

---

### 7. Awesome Flipper Zero (Resource List) ⭐⭐⭐⭐⭐

**Repository:** https://github.com/djsime1/awesome-flipperzero
**Type:** Curated list of resources
**Content:** Links to 100+ Flipper Zero resources

#### Features:
- Curated quality resources
- Organized by category
- Regularly updated
- Links to more databases

#### Use Case:** Discovery of additional databases

---

## Additional Sources (Found via Web Search)

### 8. ADolbyB Flipper Files

**Repository:** https://github.com/ADolbyB/flipper-zero-files
**Content:** Collection from various sources

### 9. magikh0e FlipperZero_Stuff

**Repository:** https://github.com/magikh0e/FlipperZero_Stuff
**Content:** IR, Sub-GHz, remotes

### 10. honeer flipper-base

**Repository:** https://github.com/honeer/flipper-base
**Description:** Database of all things Flipper Zero

### 11. spoof_temp (Synthetic Generator)

**Repository:** https://github.com/rgerganov/spoof_temp
**Type:** Temperature sensor spoofer
**Protocols:** Oregon Scientific

---

## File Format Comparison

### .cu8 Format (RTL_433 Native)

**Description:** Complex Unsigned 8-bit I/Q samples
**Channels:** 2 (I and Q)
**Sample Rate:** Typically 250kHz
**Size:** Large (raw SDR data)

**Structure:**
```
byte[0] = I sample (0-255)
byte[1] = Q sample (0-255)
byte[2] = I sample
byte[3] = Q sample
...
```

**Pros:**
- Native RTL_433 format
- Contains full signal information
- Can be reprocessed with different decoders

**Cons:**
- Large file sizes
- Requires conversion to .sub for our use case

---

### .sub Format (Flipper Zero)

**Description:** Pulse timing array
**Variants:** KEY (decoded) or RAW (pulse data)
**Size:** Small (text file)

**Structure (RAW):**
```
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 1024 -512 512 -1024 ...
```

**Pros:**
- Small file size
- Human-readable
- Ready to use with our parser

**Cons:**
- Only pulse timing (loses amplitude/frequency info)
- KEY format cannot be re-decoded

---

## Converting Between Formats

### .cu8 → .sub Conversion (TODO)

**Challenge:** RTL_433 test files are in `.cu8` format, we need `.sub`

**Required Steps:**
1. **Demodulate .cu8** → Extract pulse data
2. **Extract timing** → Pulse widths in microseconds
3. **Generate .sub** → Create Flipper format file

**Pseudo-code:**
```python
def convert_cu8_to_sub(cu8_file, output_sub):
    """Convert RTL_433 .cu8 to Flipper .sub"""
    # 1. Load .cu8 file (I/Q samples)
    iq_samples = load_cu8(cu8_file)

    # 2. Demodulate (OOK/FSK)
    demod_signal = demodulate_ook(iq_samples)

    # 3. Extract pulse timing
    pulse_data = extract_pulses(demod_signal)

    # 4. Generate .sub file
    sub_content = f"""Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: {' '.join(map(str, pulse_data))}
"""

    with open(output_sub, 'w') as f:
        f.write(sub_content)
```

**Tools Needed:**
- `scipy` for signal processing
- `numpy` for array operations
- RTL_433 source code as reference

**Priority:** Medium (would unlock 1,000+ known-good test files)

---

## Testing Strategy

### Phase 1: Quick Validation (30 minutes)

**Dataset:** FlipperZero-Subghz-DB (ready to use)

**Steps:**
1. Download FlipperZero-Subghz-DB
2. Extract weather sensor .sub files:
   ```bash
   find FlipperZero-Subghz-DB -name "*.sub" | \
     xargs grep -l "Weather\|Oregon\|Acurite\|LaCrosse" > weather.txt
   ```
3. Test with 10-20 files
4. Measure decode success rate

**Expected Result:** 10-30% success rate (most are proprietary protocols)

---

### Phase 2: High-Quality Testing (2 hours)

**Dataset:** UberGuidoZ Flipper (curated)

**Steps:**
1. Download UberGuidoZ database
2. Search for specific manufacturers:
   ```bash
   find Flipper -name "*.sub" -exec grep -l "Acurite\|Oregon" {} \;
   ```
3. Test all matches
4. Document successes/failures

**Expected Result:** 40-60% success rate (better curation)

---

### Phase 3: Gold Standard Testing (4 hours)

**Dataset:** RTL_433 test files (requires converter)

**Steps:**
1. Clone rtl_433_tests repository
2. Create .cu8 → .sub converter
3. Convert 100 test files
4. Test all converted files
5. Validate 100% decode rate

**Expected Result:** 95-100% success rate (known good signals)

---

### Phase 4: Synthetic Testing (1 hour)

**Dataset:** WSSFlipperZero (generated)

**Steps:**
1. Clone WSSFlipperZero
2. Generate 10 synthetic weather sensor .sub files
3. Test all generated files
4. Verify decode accuracy

**Expected Result:** 80-95% success rate (synthetic signals, protocol-compliant)

---

## Download Script

Created: `scripts/download_rf_test_datasets.sh`

**Usage:**
```bash
chmod +x scripts/download_rf_test_datasets.sh
./scripts/download_rf_test_datasets.sh
```

**Downloads:**
1. rtl_433_tests (1,000+ .cu8 files)
2. FlipperZero-Subghz-DB (13,717 .sub files)
3. UberGuidoZ Flipper (~10,000 files)
4. WSSFlipperZero (synthetic generators)
5. Full_Flipper_Database (comprehensive)
6. awesome-flipperzero (resource list)

**Total Download Size:** ~1-2GB
**Total Files:** 30,000+ signals

---

## Success Metrics

### By Database

| Database | Files | Format | Expected Success |
|----------|-------|--------|------------------|
| rtl_433_tests | 1,000+ | .cu8* | 95-100% |
| UberGuidoZ | 10,000+ | .sub | 40-60% |
| FlipperZero-Subghz-DB | 13,717 | .sub | 10-30% |
| WSSFlipperZero | Generated | .sub | 80-95% |

*Requires conversion to .sub

### By Protocol Type

| Protocol Category | Expected Success |
|-------------------|------------------|
| Weather Sensors (Oregon, Acurite, LaCrosse) | 90%+ |
| TPMS (Tire Pressure) | 80%+ |
| Wireless Thermometers | 85%+ |
| Security Sensors | 70%+ |
| Doorbells | 65%+ |
| Garage Door Openers | <10% |
| Gate Remotes | <5% |

---

## Next Steps

### Immediate (High Priority)

1. **Run download script** to get datasets
   ```bash
   ./scripts/download_rf_test_datasets.sh
   ```

2. **Test with FlipperZero-Subghz-DB** (no conversion needed)
   ```bash
   # Extract weather sensors
   find data/rf_test_datasets/FlipperZero-Subghz-DB -name "*.sub" | \
     xargs grep -l "Weather\|Oregon\|Acurite" > weather_sensors.txt

   # Test first 10
   head -10 weather_sensors.txt | while read file; do
       echo "Testing: $file"
       PYTHONPATH=. python3 scripts/test_rtl433_with_known_devices.py "$file"
   done
   ```

3. **Measure success rate** and document results

### Short-Term (Medium Priority)

1. **Create .cu8 → .sub converter**
2. **Convert rtl_433_tests files**
3. **Achieve 95%+ decode rate** with known-good signals

### Long-Term (Low Priority)

1. **Build curated test dataset** (100 known-good .sub files)
2. **Create automated testing pipeline**
3. **Contribute findings back to rtl_433 project**

---

## References

### Web Search Queries Used

1. `flipper zero sub files database weather sensor github`
2. `rtl_433 test files samples cu8 github protocol database`
3. `"acurite" OR "oregon scientific" OR "lacrosse" weather sensor .sub file github`

### Repositories Found

- https://github.com/merbanan/rtl_433_tests
- https://github.com/Zero-Sploit/FlipperZero-Subghz-DB
- https://github.com/UberGuidoZ/Flipper
- https://github.com/DefinetlyNotAI/Full_Flipper_Database
- https://github.com/timdigga/WSSFlipperZero
- https://github.com/timdigga/OWMFlipperZero
- https://github.com/djsime1/awesome-flipperzero
- https://github.com/ADolbyB/flipper-zero-files
- https://github.com/magikh0e/FlipperZero_Stuff
- https://github.com/honeer/flipper-base
- https://github.com/rgerganov/spoof_temp

---

## Summary

We have identified **6 primary RF signal databases** with 30,000+ labeled captures:

**Best for immediate testing:**
- FlipperZero-Subghz-DB (13,717 .sub files, ready to use)
- UberGuidoZ Flipper (10,000+ files, well-organized)

**Best for high-quality validation:**
- rtl_433_tests (1,000+ files, 100% labeled, requires conversion)

**Best for synthetic testing:**
- WSSFlipperZero (generates known-good weather sensor signals)

**Next Action:** Run `./scripts/download_rf_test_datasets.sh` to download all databases and start testing!