feat: multi-format capture ingestion (rtl_433, CSV, ZIP) + dedup
Adds src/api/ingest.py and refactors the live upload handler to auto-detect and route each submission by format instead of assuming .sub: - Flipper .sub -> existing parser + signature matcher (unchanged) - rtl_433 .json/.ndjson -> decoded; model is the device (conf 1.0) - Wigle-style .csv -> decoded; one observation per row (conf 0.9) - .zip batch -> recursed; any mix of the above Also adds stable dedup (SHA256 for whole files, composite key for decoded records) so re-submissions are skipped rather than stored twice, optional GPS privacy rounding via manifest privacy_gps_decimals, a session-level GPS fallback, and helpful errors for unsupported formats. Documented in docs/SUBMISSION_FORMAT.md with rtl_433/CSV sample fixtures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
@@ -0,0 +1,126 @@
|
||||
# GigLez Submission Format
|
||||
|
||||
GigLez accepts Sub-GHz captures from a range of popular tools. Every submission
|
||||
is a `multipart/form-data` `POST` to:
|
||||
|
||||
```
|
||||
POST /api/v1/captures/upload
|
||||
```
|
||||
|
||||
with two parts:
|
||||
|
||||
| Part | Type | Purpose |
|
||||
|------|------|---------|
|
||||
| `files` | one or more file uploads | the captures themselves |
|
||||
| `manifest` | JSON string (form field) | GPS, timestamps, session + privacy options |
|
||||
|
||||
Format is **auto-detected per file** (by extension, then by content sniffing), so
|
||||
you can mix types in a single request. Supported formats:
|
||||
|
||||
| Format | Extensions | Detected by | Handling |
|
||||
|--------|-----------|-------------|----------|
|
||||
| Flipper Zero | `.sub` | `Filetype:` header | parsed + signature-matched (category router → protocol DB) |
|
||||
| rtl_433 | `.json`, `.ndjson`, `.jsonl` | leading `{` / `[` | already decoded — model **is** the device |
|
||||
| Wigle-style tabular | `.csv` | comma in first line | already decoded — one row per observation |
|
||||
| Batch | `.zip` (`PK` magic) | zip magic | recursed; any mix of the above + optional `manifest.json` |
|
||||
|
||||
---
|
||||
|
||||
## Manifest
|
||||
|
||||
```json
|
||||
{
|
||||
"data_source": "production",
|
||||
"session_uuid": "optional-session-id",
|
||||
"latitude": 34.0200,
|
||||
"longitude": -118.3000,
|
||||
"privacy_gps_decimals": 4,
|
||||
"captures": [
|
||||
{
|
||||
"filename": "capture_001.sub",
|
||||
"latitude": 34.0201,
|
||||
"longitude": -118.3011,
|
||||
"timestamp": "2026-07-18T12:05:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Scope | Notes |
|
||||
|-------|-------|-------|
|
||||
| `data_source` | submission | `production`, `test`, or `mock`. Defaults to `production`. |
|
||||
| `session_uuid` | submission | Optional grouping id, stored as `session_id`. |
|
||||
| `latitude` / `longitude` | submission | **Session-level GPS fallback** for decoded records that carry no coords. |
|
||||
| `privacy_gps_decimals` | submission | If set, all coordinates are rounded to this many decimals before storage. Omit to store full precision. |
|
||||
| `captures[]` | per file | Keyed by `filename`. Supplies per-file GPS + timestamp. |
|
||||
|
||||
### GPS resolution order
|
||||
|
||||
1. **Coordinates in the record itself** (rtl_433 / CSV `latitude`/`longitude`, or GPS embedded in a `.sub` filename)
|
||||
2. **Per-file manifest entry** (`captures[]` matched by `filename`)
|
||||
3. **Session-level manifest GPS** (top-level `latitude`/`longitude`)
|
||||
|
||||
The chosen source is recorded on each capture as `gps_source`
|
||||
(`record` / `manifest` / `session` / `manual` / `none`).
|
||||
|
||||
---
|
||||
|
||||
## Format details
|
||||
|
||||
### Flipper Zero `.sub`
|
||||
Both decoded key files (`Protocol` / `Bit` / `Key`) and RAW pulse captures
|
||||
(`RAW_Data`). These are matched through the category router and protocol
|
||||
database; the result carries `match_confidence` and `match_method`.
|
||||
|
||||
### rtl_433 `.json` / `.ndjson`
|
||||
One JSON object per line (NDJSON) **or** a single JSON array. rtl_433 output is
|
||||
already decoded, so the reported `model` is taken as the device (no signature
|
||||
matching). Recognised fields include `model`, `id`, `channel`, `battery_ok`,
|
||||
`temperature_C`, `humidity`, `pressure_kPa`, `mod`, `freq`, `time`. Frequency in
|
||||
MHz is normalised to Hz. `match_method` = `rtl_433_decoded`, confidence `1.0`.
|
||||
|
||||
### Wigle-style `.csv`
|
||||
Header row + one observation per row. Useful columns: `model` (or
|
||||
`device_name` / `protocol`), `manufacturer`/`brand`, `frequency` (Hz or MHz),
|
||||
`latitude`/`longitude`, `timestamp`, plus optional decoded fields. `match_method`
|
||||
= `csv_import`, confidence `0.9`.
|
||||
|
||||
### `.zip` batch
|
||||
Any mix of the above. Directory entries, dotfiles, and a top-level
|
||||
`manifest.json` member are skipped; every other member is ingested by its own
|
||||
detected format.
|
||||
|
||||
---
|
||||
|
||||
## Deduplication
|
||||
|
||||
Re-submitting the same data is a no-op. Each stored capture carries a stable
|
||||
`dedup_key`:
|
||||
|
||||
- **Whole-file formats (`.sub`)** — SHA256 of the raw file bytes.
|
||||
- **Decoded records (rtl_433 / CSV)** — composite of
|
||||
`protocol · device_name · frequency · id · channel · rounded lat/lon · timestamp` (to the second).
|
||||
|
||||
On upload, records whose key already exists are reported under `skipped` rather
|
||||
than stored again.
|
||||
|
||||
---
|
||||
|
||||
## Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Processed 9 records (0 duplicates skipped, 0 failed)",
|
||||
"successful": [ /* stored capture records */ ],
|
||||
"skipped": [ /* duplicate records, not stored */ ],
|
||||
"failed": [ { "filename": "...", "error": "..." } ],
|
||||
"total_uploaded": 2,
|
||||
"total_successful": 9,
|
||||
"total_skipped": 0,
|
||||
"total_failed": 0
|
||||
}
|
||||
```
|
||||
|
||||
A single uploaded file (rtl_433/CSV/ZIP) can expand into many capture records,
|
||||
so counts are per-record, not per-file.
|
||||
Reference in New Issue
Block a user