Files
giglez/docs/SUBMISSION_FORMAT.md
T
leetcrypt 11d8957a49 feat: multi-format capture ingestion (rtl_433, CSV, ZIP) + dedup
Adds src/api/ingest.py and refactors the live upload handler to auto-detect
and route each submission by format instead of assuming .sub:

  - Flipper .sub          -> existing parser + signature matcher (unchanged)
  - rtl_433 .json/.ndjson -> decoded; model is the device (conf 1.0)
  - Wigle-style .csv      -> decoded; one observation per row (conf 0.9)
  - .zip batch            -> recursed; any mix of the above

Also adds stable dedup (SHA256 for whole files, composite key for decoded
records) so re-submissions are skipped rather than stored twice, optional
GPS privacy rounding via manifest privacy_gps_decimals, a session-level GPS
fallback, and helpful errors for unsupported formats. Documented in
docs/SUBMISSION_FORMAT.md with rtl_433/CSV sample fixtures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-18 20:08:39 -07:00

4.4 KiB

GigLez Submission Format

GigLez accepts Sub-GHz captures from a range of popular tools. Every submission is a multipart/form-data POST to:

POST /api/v1/captures/upload

with two parts:

Part Type Purpose
files one or more file uploads the captures themselves
manifest JSON string (form field) GPS, timestamps, session + privacy options

Format is auto-detected per file (by extension, then by content sniffing), so you can mix types in a single request. Supported formats:

Format Extensions Detected by Handling
Flipper Zero .sub Filetype: header parsed + signature-matched (category router → protocol DB)
rtl_433 .json, .ndjson, .jsonl leading { / [ already decoded — model is the device
Wigle-style tabular .csv comma in first line already decoded — one row per observation
Batch .zip (PK magic) zip magic recursed; any mix of the above + optional manifest.json

Manifest

{
  "data_source": "production",
  "session_uuid": "optional-session-id",
  "latitude": 34.0200,
  "longitude": -118.3000,
  "privacy_gps_decimals": 4,
  "captures": [
    {
      "filename": "capture_001.sub",
      "latitude": 34.0201,
      "longitude": -118.3011,
      "timestamp": "2026-07-18T12:05:00Z"
    }
  ]
}
Field Scope Notes
data_source submission production, test, or mock. Defaults to production.
session_uuid submission Optional grouping id, stored as session_id.
latitude / longitude submission Session-level GPS fallback for decoded records that carry no coords.
privacy_gps_decimals submission If set, all coordinates are rounded to this many decimals before storage. Omit to store full precision.
captures[] per file Keyed by filename. Supplies per-file GPS + timestamp.

GPS resolution order

  1. Coordinates in the record itself (rtl_433 / CSV latitude/longitude, or GPS embedded in a .sub filename)
  2. Per-file manifest entry (captures[] matched by filename)
  3. Session-level manifest GPS (top-level latitude/longitude)

The chosen source is recorded on each capture as gps_source (record / manifest / session / manual / none).


Format details

Flipper Zero .sub

Both decoded key files (Protocol / Bit / Key) and RAW pulse captures (RAW_Data). These are matched through the category router and protocol database; the result carries match_confidence and match_method.

rtl_433 .json / .ndjson

One JSON object per line (NDJSON) or a single JSON array. rtl_433 output is already decoded, so the reported model is taken as the device (no signature matching). Recognised fields include model, id, channel, battery_ok, temperature_C, humidity, pressure_kPa, mod, freq, time. Frequency in MHz is normalised to Hz. match_method = rtl_433_decoded, confidence 1.0.

Wigle-style .csv

Header row + one observation per row. Useful columns: model (or device_name / protocol), manufacturer/brand, frequency (Hz or MHz), latitude/longitude, timestamp, plus optional decoded fields. match_method = csv_import, confidence 0.9.

.zip batch

Any mix of the above. Directory entries, dotfiles, and a top-level manifest.json member are skipped; every other member is ingested by its own detected format.


Deduplication

Re-submitting the same data is a no-op. Each stored capture carries a stable dedup_key:

  • Whole-file formats (.sub) — SHA256 of the raw file bytes.
  • Decoded records (rtl_433 / CSV) — composite of protocol · device_name · frequency · id · channel · rounded lat/lon · timestamp (to the second).

On upload, records whose key already exists are reported under skipped rather than stored again.


Response

{
  "success": true,
  "message": "Processed 9 records (0 duplicates skipped, 0 failed)",
  "successful": [ /* stored capture records */ ],
  "skipped":    [ /* duplicate records, not stored */ ],
  "failed":     [ { "filename": "...", "error": "..." } ],
  "total_uploaded": 2,
  "total_successful": 9,
  "total_skipped": 0,
  "total_failed": 0
}

A single uploaded file (rtl_433/CSV/ZIP) can expand into many capture records, so counts are per-record, not per-file.