leetcrypt
11d8957a49
Adds src/api/ingest.py and refactors the live upload handler to auto-detect and route each submission by format instead of assuming .sub: - Flipper .sub -> existing parser + signature matcher (unchanged) - rtl_433 .json/.ndjson -> decoded; model is the device (conf 1.0) - Wigle-style .csv -> decoded; one observation per row (conf 0.9) - .zip batch -> recursed; any mix of the above Also adds stable dedup (SHA256 for whole files, composite key for decoded records) so re-submissions are skipped rather than stored twice, optional GPS privacy rounding via manifest privacy_gps_decimals, a session-level GPS fallback, and helpful errors for unsupported formats. Documented in docs/SUBMISSION_FORMAT.md with rtl_433/CSV sample fixtures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
127 lines
4.4 KiB
Markdown
127 lines
4.4 KiB
Markdown
# GigLez Submission Format
|
|
|
|
GigLez accepts Sub-GHz captures from a range of popular tools. Every submission
|
|
is a `multipart/form-data` `POST` to:
|
|
|
|
```
|
|
POST /api/v1/captures/upload
|
|
```
|
|
|
|
with two parts:
|
|
|
|
| Part | Type | Purpose |
|
|
|------|------|---------|
|
|
| `files` | one or more file uploads | the captures themselves |
|
|
| `manifest` | JSON string (form field) | GPS, timestamps, session + privacy options |
|
|
|
|
Format is **auto-detected per file** (by extension, then by content sniffing), so
|
|
you can mix types in a single request. Supported formats:
|
|
|
|
| Format | Extensions | Detected by | Handling |
|
|
|--------|-----------|-------------|----------|
|
|
| Flipper Zero | `.sub` | `Filetype:` header | parsed + signature-matched (category router → protocol DB) |
|
|
| rtl_433 | `.json`, `.ndjson`, `.jsonl` | leading `{` / `[` | already decoded — model **is** the device |
|
|
| Wigle-style tabular | `.csv` | comma in first line | already decoded — one row per observation |
|
|
| Batch | `.zip` (`PK` magic) | zip magic | recursed; any mix of the above + optional `manifest.json` |
|
|
|
|
---
|
|
|
|
## Manifest
|
|
|
|
```json
|
|
{
|
|
"data_source": "production",
|
|
"session_uuid": "optional-session-id",
|
|
"latitude": 34.0200,
|
|
"longitude": -118.3000,
|
|
"privacy_gps_decimals": 4,
|
|
"captures": [
|
|
{
|
|
"filename": "capture_001.sub",
|
|
"latitude": 34.0201,
|
|
"longitude": -118.3011,
|
|
"timestamp": "2026-07-18T12:05:00Z"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
| Field | Scope | Notes |
|
|
|-------|-------|-------|
|
|
| `data_source` | submission | `production`, `test`, or `mock`. Defaults to `production`. |
|
|
| `session_uuid` | submission | Optional grouping id, stored as `session_id`. |
|
|
| `latitude` / `longitude` | submission | **Session-level GPS fallback** for decoded records that carry no coords. |
|
|
| `privacy_gps_decimals` | submission | If set, all coordinates are rounded to this many decimals before storage. Omit to store full precision. |
|
|
| `captures[]` | per file | Keyed by `filename`. Supplies per-file GPS + timestamp. |
|
|
|
|
### GPS resolution order
|
|
|
|
1. **Coordinates in the record itself** (rtl_433 / CSV `latitude`/`longitude`, or GPS embedded in a `.sub` filename)
|
|
2. **Per-file manifest entry** (`captures[]` matched by `filename`)
|
|
3. **Session-level manifest GPS** (top-level `latitude`/`longitude`)
|
|
|
|
The chosen source is recorded on each capture as `gps_source`
|
|
(`record` / `manifest` / `session` / `manual` / `none`).
|
|
|
|
---
|
|
|
|
## Format details
|
|
|
|
### Flipper Zero `.sub`
|
|
Both decoded key files (`Protocol` / `Bit` / `Key`) and RAW pulse captures
|
|
(`RAW_Data`). These are matched through the category router and protocol
|
|
database; the result carries `match_confidence` and `match_method`.
|
|
|
|
### rtl_433 `.json` / `.ndjson`
|
|
One JSON object per line (NDJSON) **or** a single JSON array. rtl_433 output is
|
|
already decoded, so the reported `model` is taken as the device (no signature
|
|
matching). Recognised fields include `model`, `id`, `channel`, `battery_ok`,
|
|
`temperature_C`, `humidity`, `pressure_kPa`, `mod`, `freq`, `time`. Frequency in
|
|
MHz is normalised to Hz. `match_method` = `rtl_433_decoded`, confidence `1.0`.
|
|
|
|
### Wigle-style `.csv`
|
|
Header row + one observation per row. Useful columns: `model` (or
|
|
`device_name` / `protocol`), `manufacturer`/`brand`, `frequency` (Hz or MHz),
|
|
`latitude`/`longitude`, `timestamp`, plus optional decoded fields. `match_method`
|
|
= `csv_import`, confidence `0.9`.
|
|
|
|
### `.zip` batch
|
|
Any mix of the above. Directory entries, dotfiles, and a top-level
|
|
`manifest.json` member are skipped; every other member is ingested by its own
|
|
detected format.
|
|
|
|
---
|
|
|
|
## Deduplication
|
|
|
|
Re-submitting the same data is a no-op. Each stored capture carries a stable
|
|
`dedup_key`:
|
|
|
|
- **Whole-file formats (`.sub`)** — SHA256 of the raw file bytes.
|
|
- **Decoded records (rtl_433 / CSV)** — composite of
|
|
`protocol · device_name · frequency · id · channel · rounded lat/lon · timestamp` (to the second).
|
|
|
|
On upload, records whose key already exists are reported under `skipped` rather
|
|
than stored again.
|
|
|
|
---
|
|
|
|
## Response
|
|
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Processed 9 records (0 duplicates skipped, 0 failed)",
|
|
"successful": [ /* stored capture records */ ],
|
|
"skipped": [ /* duplicate records, not stored */ ],
|
|
"failed": [ { "filename": "...", "error": "..." } ],
|
|
"total_uploaded": 2,
|
|
"total_successful": 9,
|
|
"total_skipped": 0,
|
|
"total_failed": 0
|
|
}
|
|
```
|
|
|
|
A single uploaded file (rtl_433/CSV/ZIP) can expand into many capture records,
|
|
so counts are per-record, not per-file.
|