Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research(accuracy): rtl_433 IQ augmentation — coverage/precision tradeoff, not shipped

Browse Source 
Adds reproducible tooling to test whether genuinely-independent rtl_433 test
captures can relieve the statistical classifier's minority-class starvation
(train: Weather 12, Security 2). Local Flipper corpora are byte-identical
mirrors, so rtl_433 is the only independent on-disk source.

- rtl433_iq_demod.py: .cu8 (interleaved uint8 IQ) -> Flipper-style signed µs
  pulse train via amplitude/OOK demod. A quality-gate self-rejects FSK/degenerate
  captures. Recovered pulse widths match rtl_433's own -A analysis.
- experiment_rtl433_augment.py: conservative folder->category map + an HONEST
  evaluation — train on Flipper (group-disjoint) +/- rtl_433, score on held-out
  FLIPPER (regression guardrail) and held-out rtl_433 (new capability).

RESULT: adding rtl_433 to training DEGRADES the Flipper gate metric
(balanced 0.625 -> 0.412), collapsing Remote Control (20/26 -> 7-10/26) as
cross-domain samples bleed into Flipper's remote region and their volume swamps
it. Small capped adds keep Flipper within seed-noise while gaining large
rtl_433-domain Weather recognition — a coverage-vs-precision tradeoff, not a
clean win. Per gate-metric discipline the production model is NOT regenerated;
the tooling is kept to re-measure in-domain once real Flipper weather/security
captures arrive via the platform.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-07-19 20:13:14 -07:00
parent 7535feb445
commit 3aadc09e13
2 changed files with 353 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/experiment_rtl433_augment.py
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""
Honest experiment: does rtl_433 IQ data help the category classifier?
=====================================================================
The statistical category classifier is starved on minority classes
(train counts ~ Doorbell 39, Weather 12, Security 2). Local Flipper .sub
corpora are byte-identical mirrors, so they add nothing. The rtl_433 project
ships genuinely-independent labelled OOK captures (``.cu8`` IQ), heavy on
exactly the starved classes (weather + PIR/door/smoke security).
This script measures — honestly — whether adding those demodulated captures to
TRAINING helps, without fooling ourselves:
* Flipper captures are split group-disjoint into train/test (the real
product distribution; prod input is Flipper .sub RAW).
* rtl_433 captures are demodulated (``rtl433_iq_demod.demod_file``), split
group-disjoint by device folder, and used to AUGMENT training only.
* We compare, on the SAME held-out FLIPPER test set:
baseline = train on Flipper-only
augmented = train on Flipper + rtl_433
Regression check: do the well-populated Flipper classes
(Garage/Remote/Fan/Doorbell) get WORSE? If augmentation tanks them, reject.
* New-capability check: on a held-out rtl_433 test set, can the augmented
model recognise Weather/Security at all (classes Flipper can't measure —
too few held-out samples)?
Nothing here touches the serving path; it only decides whether to retrain.
"""
from __future__ import annotations
import random
import warnings
from collections import Counter
from pathlib import Path
import numpy as np
warnings.filterwarnings("ignore")
from scripts.rtl433_iq_demod import demod_file
from scripts.train_category_classifier import (
collect as collect_flipper, DEFAULT_DATASET, FEATURE_NAMES, extract_features,
)
from src.matcher.timing_analyzer import get_timing_analyzer
from src.matcher.preamble_detector import get_preamble_detector
RTL433_ROOT = (Path(__file__).parent.parent
/ "data/rf_test_datasets/rtl_433_tests/tests")
# Conservative folder -> category map. Only clearly in-taxonomy devices; the
# demod quality-gate additionally drops any FSK/degenerate captures. TPMS,
# energy/utility meters, thermostats and unknowns are intentionally omitted.
RTL433_FOLDER_TO_CATEGORY = {
# Weather Sensor (the biggest starved-class win)
**{f: "Weather Sensor" for f in [
"acurite", "alectov1", "alecto_ws_1200", "ambient_weather", "auriol",
"bresser_3ch", "bresser_5in1", "bresser_6in1", "bt_rain", "calibeur",
"companion_wtr001", "conrad_pool_thermometer", "cotech", "cresta_ws688",
"ecowitt", "EcoWitt-WH40", "esperanza_ews", "Eurochron-EFTH800",
"eurochron-th", "fineoffset", "froggit_wh1080_Pass14c", "ft004b",
"hideki", "holman_ws5029", "imagintronix_wh5", "infactory_PV-8796-675",
"InFactroy-Temp-Humidity-Sensor-T05K-THC", "inkbird", "inovalley-kw9015b",
"lacrosse", "lacrosse_ltv", "lacrosse_ws7000", "maverick_et-73",
"maverick_et733", "mebus_te204nl", "misol", "missil_ml0757", "nexus",
"Opus-XT300", "oregon_scientific", "prologue", "proove", "rubicson",
"s3318p", "sharp_spc344", "sharp_spc775", "solight_te44", "springfield",
"TFA_30.3196_TempHumid", "tfa_30_3211_02", "TFA-Drop-30.3233.01",
"TFA_Marbella", "TFA-Pool-thermometer-30.3160", "TFA-Twin-Plus-30.3049",
"thermopro-tp11", "thermopro-tp12", "thermopro-tx2", "TS-FT002",
"ttx201", "tx22-it", "wssensor", "wt0124", "XC-0324", "xc0348",
"Globaltronics", "PoolWT0122", "rh787t",
]},
# Security Sensor (motion / PIR / smoke / door / alarm)
**{f: "Security Sensor" for f in [
"generic_door_sensor", "generic_motion", "honeywell", "honeywell_5816",
"honeywell_5890PI", "honeywell_activlink", "dsc", "simplisafe",
"skylink_motion", "smoke_gs558", "cavius", "KIDDE_RF-SM-ACDC",
"visonic_powercode", "interlogix", "x10_sec", "EV1527-PIR-SGOOWAY",
"Kerui-D026", "chuango",
]},
# Doorbell
**{f: "Doorbell" for f in [
"Byron-BY101", "Byron-BY34", "Byron_db304", "door_bell", "quhwa",
]},
# Remote Control (remotes, outlet switches, key fobs)
**{f: "Remote Control" for f in [
"akhan_remote", "dish_remote_6.3", "directv", "EV1527-Universal-Remote",
"fordremote", "generic-4ch-black-remote-315mhz",
"generic-4ch-black-remote-433mhz", "generic_remote",
"ge-coloreffects-remote", "honda_remote", "HT680_remote",
"hyundai_remote", "intertechno", "nexa_LMST-606", "newkaku", "PT2262",
"rayrun_rm03", "waveman", "x10", "philips", "Philips-AJ7010",
"status-rc202", "kangtai", "kedsum", "etekcity", "elro", "blyss",
"brennstuhl_rcs_2044", "telldus", "smarthome", "silvercrest",
"uni-com-66125", "valeo",
]},
# Garage Door Opener / gate
**{f: "Garage Door Opener" for f in [
"cardin", "LiftMaster_4330E", "secplus", "keeloq", "Microchip-HCS200",
]},
# Fan Controller
**{f: "Fan Controller" for f in [
"lucci-air-ceiling-fan-remote", "SQMLtdFan",
]},
}
def collect_rtl433(seed: int, per_folder: int = 40):
"""Demodulate rtl_433 captures into (X, y, groups) using the folder map."""
rng = random.Random(seed)
ta, pd = get_timing_analyzer(), get_preamble_detector()
X, y, groups = [], [], []
skipped = Counter()
for folder, category in RTL433_FOLDER_TO_CATEGORY.items():
fpath = RTL433_ROOT / folder
if not fpath.is_dir():
skipped["missing_folder"] += 1
continue
cu8s = sorted(fpath.rglob("*.cu8"))
rng.shuffle(cu8s)
taken = 0
for c in cu8s:
if taken >= per_folder:
break
pulses, freq = demod_file(c)
if pulses is None:
skipped["rejected_demod"] += 1
continue
feats = extract_features(pulses, freq, ta, pd)
if feats is None:
skipped["no_timing"] += 1
continue
X.append(feats)
y.append(category)
groups.append(f"rtl433:{folder}") # group per device folder
taken += 1
return np.array(X), np.array(y), np.array(groups), skipped
def per_class_recall(y_true, y_pred, labels):
out = {}
for lab in labels:
idx = y_true == lab
n = int(idx.sum())
out[lab] = (int((y_pred[idx] == lab).sum()), n)
return out
def main():
from sklearn.model_selection import GroupShuffleSplit
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import balanced_accuracy_score, accuracy_score
seed = 42
print("Collecting Flipper RAW ...")
Xf, yf, gf, ff, _ = collect_flipper(DEFAULT_DATASET, 400, seed)
print(f" Flipper: {len(Xf)} samples {dict(Counter(yf))}")
print("Demodulating rtl_433 IQ ...")
Xr, yr, gr, sk = collect_rtl433(seed)
print(f" rtl_433: {len(Xr)} samples {dict(Counter(yr))} (skipped {dict(sk)})")
labels = sorted(set(yf) | set(yr))
# Group-disjoint Flipper train/test (the real product distribution).
gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=seed)
ftr, fte = next(gss.split(Xf, yf, gf))
Xf_tr, Xf_te = Xf[ftr], Xf[fte]
yf_tr, yf_te = yf[ftr], yf[fte]
# Group-disjoint rtl_433 train/test.
rtr, rte = next(GroupShuffleSplit(1, test_size=0.30, random_state=seed)
.split(Xr, yr, gr))
Xr_tr, Xr_te = Xr[rtr], Xr[rte]
yr_tr, yr_te = yr[rtr], yr[rte]
def fit(X, y):
return GradientBoostingClassifier(random_state=seed).fit(X, y)
base = fit(Xf_tr, yf_tr)
aug = fit(np.vstack([Xf_tr, Xr_tr]), np.concatenate([yf_tr, yr_tr]))
print("\n" + "=" * 70)
print("A) REGRESSION CHECK — held-out FLIPPER test (does augmentation hurt?)")
print("=" * 70)
for name, model in [("baseline (Flipper-only)", base),
("augmented (Flipper+rtl433)", aug)]:
p = model.predict(Xf_te)
print(f"\n{name}: bal={balanced_accuracy_score(yf_te, p):.3f} "
f"top1={accuracy_score(yf_te, p):.3f}")
for lab, (ok, n) in per_class_recall(yf_te, p, labels).items():
if n:
print(f" {lab:20} {ok:3d}/{n:<3d}")
print("\n" + "=" * 70)
print("B) NEW-CAPABILITY — held-out rtl_433 test (classes Flipper can't measure)")
print("=" * 70)
for name, model in [("baseline (Flipper-only)", base),
("augmented (Flipper+rtl433)", aug)]:
p = model.predict(Xr_te)
print(f"\n{name}: bal={balanced_accuracy_score(yr_te, p):.3f} "
f"top1={accuracy_score(yr_te, p):.3f}")
for lab, (ok, n) in per_class_recall(yr_te, p, labels).items():
if n:
print(f" {lab:20} {ok:3d}/{n:<3d}")
if __name__ == "__main__":
main()
scripts/rtl433_iq_demod.py
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""
rtl_433 .cu8 IQ -> Flipper-style signed pulse train (µs)
==========================================================
Turns the rtl_433 project's ``.cu8`` test captures (interleaved I/Q uint8) into
the same signed-duration pulse array our RAW feature extractor already consumes
(``+high`` / ``-low`` in microseconds), so genuinely-independent, non-Flipper
captures can augment the statistical category classifier's training set.
Why amplitude (OOK) demod: the target Sub-GHz device classes we care about
(weather sensors, doorbells, PIR/door/smoke security sensors, simple remotes)
are overwhelmingly OOK/ASK — the carrier is keyed on/off, so the signal
*envelope* (magnitude of the complex sample) is the message. FSK captures keep
constant amplitude, so this demod yields no usable transitions on them; that is
a feature, not a bug — ``demod_file`` returns ``None`` and such captures
self-filter out of the training set via ``looks_like_ook``.
The sample rate and centre frequency are read from the rtl_433 filename
convention ``*_<freq>M_<rate>k.cu8`` (e.g. ``g003_433.92M_250k.cu8``), falling
back to 250 kHz / 433.92 MHz.
This is a training-data-prep tool (offline), not part of the serving path.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import List, Optional, Tuple
import numpy as np
DEFAULT_RATE = 250_000
DEFAULT_FREQ = 433_920_000
# Quality gate — reject captures that did not demodulate into a plausible
# OOK pulse train (FSK, pure noise, or empty).
MIN_TRANSITIONS = 16 # need real structure, not one long burst
MIN_SHORT_US = 40 # ignore sub-symbol glitches
MAX_KEEP_PULSES = 1024 # cap runaway repeats (Flipper captures are bounded)
def parse_rate_freq(name: str) -> Tuple[int, int]:
"""Extract (sample_rate_hz, frequency_hz) from an rtl_433 filename."""
rate_m = re.search(r"_(\d+)k", name)
rate = int(rate_m.group(1)) * 1000 if rate_m else DEFAULT_RATE
freq_m = re.search(r"_(\d+(?:\.\d+)?)M", name)
freq = int(float(freq_m.group(1)) * 1_000_000) if freq_m else DEFAULT_FREQ
return rate, freq
def _magnitude(path: Path) -> Optional[np.ndarray]:
raw = np.fromfile(path, dtype=np.uint8)
if raw.size < 4:
return None
raw = raw.astype(np.float32) - 127.5 # centre uint8 IQ
i, q = raw[0::2], raw[1::2]
n = min(i.size, q.size)
if n == 0:
return None
return np.sqrt(i[:n] * i[:n] + q[:n] * q[:n])
def _runs_to_pulses(hyst: np.ndarray, us_per_sample: float) -> List[int]:
"""Run-length encode a boolean high/low mask into signed µs durations."""
if hyst.size == 0:
return []
# Indices where the level changes.
change = np.flatnonzero(np.diff(hyst.view(np.int8))) + 1
bounds = np.concatenate(([0], change, [hyst.size]))
pulses: List[int] = []
for a, b in zip(bounds[:-1], bounds[1:]):
dur = (b - a) * us_per_sample
if dur < MIN_SHORT_US:
continue
sign = 1 if hyst[a] else -1
pulses.append(int(round(dur)) * sign)
return pulses
def _trim_gaps(pulses: List[int]) -> List[int]:
"""Drop leading/trailing low (gap) runs so the train starts on a pulse."""
start = 0
while start < len(pulses) and pulses[start] < 0:
start += 1
end = len(pulses)
while end > start and pulses[end - 1] < 0:
end -= 1
return pulses[start:end]
def demod_file(path: Path) -> Tuple[Optional[List[int]], int]:
"""Demodulate one .cu8 file to (pulse_train | None, frequency_hz)."""
path = Path(path)
rate, freq = parse_rate_freq(path.name)
us_per_sample = 1_000_000.0 / rate
mag = _magnitude(path)
if mag is None or mag.size == 0:
return None, freq
# Threshold between the noise floor and the signal peak. OOK captures have
# a large high/low separation, so a fixed fraction of the range is robust;
# hysteresis is unnecessary given the >15 dB SNR of these test files.
lo = float(np.percentile(mag, 50))
hi = float(np.percentile(mag, 99))
if hi - lo < 1.0: # no amplitude modulation (FSK / dead air)
return None, freq
thr = lo + 0.3 * (hi - lo)
hyst = mag > thr
pulses = _trim_gaps(_runs_to_pulses(hyst, us_per_sample))
if len(pulses) > MAX_KEEP_PULSES:
pulses = pulses[:MAX_KEEP_PULSES]
if not looks_like_ook(pulses):
return None, freq
return pulses, freq
def looks_like_ook(pulses: List[int]) -> bool:
"""True when the train has enough OOK structure to be a real signal."""
if pulses is None or len(pulses) < MIN_TRANSITIONS:
return False
highs = [p for p in pulses if p > 0]
if len(highs) < MIN_TRANSITIONS // 2:
return False
# A single constant width with no variation is not a keyed message.
return len(set(highs)) > 1
if __name__ == "__main__":
import argparse
ap = argparse.ArgumentParser(description="Demodulate an rtl_433 .cu8 to a pulse train")
ap.add_argument("file", type=Path)
args = ap.parse_args()
pulses, freq = demod_file(args.file)
if pulses is None:
print(f"{args.file.name}: no OOK pulse train (FSK/degenerate)")
else:
print(f"{args.file.name}: freq={freq} pulses={len(pulses)}")
print("first 24:", pulses[:24])