Initial commit: Phase 1 & Phase 2 infrastructure complete
This commit is contained in:
@@ -0,0 +1,389 @@
|
||||
# GigLez Architecture Decisions (Based on Wigle Analysis)
|
||||
|
||||
## Overview
|
||||
|
||||
This document captures key architectural decisions for GigLez, informed by analysis of the Wigle Android wardriving client.
|
||||
|
||||
**Date**: 2026-01-12
|
||||
**Status**: Proposed
|
||||
**Source**: Wigle Analysis (/docs/wigle_analysis.md)
|
||||
|
||||
---
|
||||
|
||||
## Decision 1: Database Schema - PostgreSQL with PostGIS
|
||||
|
||||
**Context**: Need to store millions of RF captures with GPS coordinates for geospatial queries.
|
||||
|
||||
**Decision**: Use PostgreSQL + PostGIS with a 3-table structure:
|
||||
- `captures` - Primary RF file data (like Wigle's `network` table)
|
||||
- `capture_matches` - Device identification results (like Wigle's `location` table)
|
||||
- `devices` - Known device signature database (reference data)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's 3-table design (network/location/route) has scaled to **349M WiFi networks** and **billions of observations**
|
||||
- PostGIS provides efficient geospatial indexing for radius/bounding box queries
|
||||
- Separate matches table allows many-to-many relationships (one capture → multiple device matches)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- MongoDB: No ACID guarantees, weaker geospatial query performance
|
||||
- Single table: Would duplicate device info, slower queries
|
||||
|
||||
**Implementation**:
|
||||
```sql
|
||||
CREATE TABLE captures (
|
||||
file_hash text primary key,
|
||||
latitude double precision not null,
|
||||
longitude double precision not null,
|
||||
geom geometry(Point, 4326), -- PostGIS spatial index
|
||||
timestamp bigint not null,
|
||||
frequency integer not null,
|
||||
protocol text,
|
||||
session_id text
|
||||
);
|
||||
CREATE INDEX idx_captures_geom ON captures USING GIST(geom);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 2: File Deduplication - SHA256 Hash as Primary Key
|
||||
|
||||
**Context**: Prevent duplicate .sub file submissions (same file uploaded multiple times).
|
||||
|
||||
**Decision**: Use SHA256 hash of file contents as primary key in `captures` table.
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses BSSID (MAC address) as primary key for natural deduplication
|
||||
- SHA256 provides **cryptographically strong** uniqueness guarantee
|
||||
- Automatic deduplication at database level (INSERT fails on duplicate)
|
||||
- No need for separate deduplication logic
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Auto-increment ID + separate hash check: More complex, requires application-level logic
|
||||
- GPS + timestamp composite key: Could have legitimate duplicates
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
import hashlib
|
||||
|
||||
def compute_file_hash(file_path: str) -> str:
|
||||
sha256 = hashlib.sha256()
|
||||
with open(file_path, 'rb') as f:
|
||||
for chunk in iter(lambda: f.read(8192), b''):
|
||||
sha256.update(chunk)
|
||||
return sha256.hexdigest()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 3: Upload Format - JSON Manifest + Binary Files
|
||||
|
||||
**Context**: Need to upload .sub files with GPS metadata to server.
|
||||
|
||||
**Decision**: Use multipart/form-data with:
|
||||
- JSON manifest (GPS coordinates, metadata)
|
||||
- Binary .sub files (unmodified)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses CSV for structured data export - works for text but not binary files
|
||||
- JSON provides structured metadata (per-file GPS, session info)
|
||||
- Binary files preserve exact capture data (no encoding issues)
|
||||
- Supports batch uploads naturally (multiple files in one request)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- CSV with base64-encoded files: Bloated (33% overhead), parsing complexity
|
||||
- Separate API calls per file: Network inefficiency, no atomic batch
|
||||
|
||||
**Implementation**:
|
||||
```json
|
||||
{
|
||||
"version": "GigLez-1.0",
|
||||
"session_id": "uuid",
|
||||
"captures": [
|
||||
{
|
||||
"file": "capture_001.sub",
|
||||
"sha256": "abc123...",
|
||||
"gps": {
|
||||
"latitude": 40.7128,
|
||||
"longitude": -74.0060,
|
||||
"accuracy": 5.0,
|
||||
"timestamp": "2025-01-11T20:30:00Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 4: Background Processing - Async Task Queue
|
||||
|
||||
**Context**: .sub file parsing and device matching are CPU-intensive operations.
|
||||
|
||||
**Decision**: Use async background task queue (FastAPI BackgroundTasks or Celery/RQ) for:
|
||||
- File parsing
|
||||
- Device signature matching
|
||||
- Database writes
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **background thread** for all database writes (THREAD_PRIORITY_BACKGROUND)
|
||||
- Prevents API timeouts on large uploads
|
||||
- Enables **immediate response** to user (202 Accepted)
|
||||
- Allows **horizontal scaling** of workers
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Synchronous processing: Slow, API timeouts on large files
|
||||
- Lambda functions: Cold start overhead, cost at scale
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from fastapi import BackgroundTasks
|
||||
|
||||
@app.post("/api/submit")
|
||||
async def submit_capture(file: UploadFile, background_tasks: BackgroundTasks):
|
||||
file_path = await save_upload(file)
|
||||
task_id = uuid4()
|
||||
|
||||
background_tasks.add_task(
|
||||
process_capture_pipeline,
|
||||
file_path=file_path,
|
||||
task_id=task_id
|
||||
)
|
||||
|
||||
return {"status": "accepted", "task_id": str(task_id)}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 5: GPS Validation - Multi-Level Checks
|
||||
|
||||
**Context**: GPS coordinates can be invalid (null island, extreme accuracy, out of bounds).
|
||||
|
||||
**Decision**: Implement Wigle's multi-level GPS validation:
|
||||
1. Bounds check (-90≤lat≤90, -180≤lon≤180)
|
||||
2. Accuracy threshold (reject if >100m)
|
||||
3. Null island check (reject lat=0, lon=0)
|
||||
4. Timestamp validation (reject if 0)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's horribleGps() function filters out **GPS noise** from hardware bugs
|
||||
- Prevents database pollution with junk coordinates
|
||||
- 100m threshold balances accuracy vs. rejection rate
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Accept all GPS: Database pollution, poor map quality
|
||||
- Strict threshold (<10m): Too restrictive, rejects valid mobile captures
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
def validate_gps(lat, lon, accuracy=None, timestamp=None):
|
||||
if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
|
||||
raise ValueError("GPS out of bounds")
|
||||
if accuracy and accuracy > 100:
|
||||
raise ValueError(f"GPS accuracy too low: {accuracy}m")
|
||||
if lat == 0.0 and lon == 0.0:
|
||||
raise ValueError("GPS at null island")
|
||||
if timestamp == 0:
|
||||
raise ValueError("Invalid timestamp")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 6: Upload Marker System - Incremental Sync
|
||||
|
||||
**Context**: Users may upload same database multiple times (after adding new captures).
|
||||
|
||||
**Decision**: Track last uploaded capture ID per user (like Wigle's PREF_DB_MARKER).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's marker system enables **incremental sync** (only new data)
|
||||
- Prevents re-uploading entire database on each sync
|
||||
- Enables **resume after failure** (upload only remaining captures)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Always upload all captures: Wasteful bandwidth, slow
|
||||
- Client-side tracking only: Lost on app reinstall
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
class UploadMarker(Base):
|
||||
__tablename__ = 'upload_markers'
|
||||
user_id = Column(String, primary_key=True)
|
||||
last_uploaded_id = Column(Integer, default=0)
|
||||
last_upload_time = Column(BigInteger)
|
||||
|
||||
def get_unuploaded_captures(user_id):
|
||||
marker = db.query(UploadMarker).filter_by(user_id=user_id).first()
|
||||
last_id = marker.last_uploaded_id if marker else 0
|
||||
|
||||
return db.query(Capture).filter(
|
||||
Capture.id > last_id,
|
||||
Capture.uploaded == False
|
||||
).all()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 7: Session Management - Run ID System
|
||||
|
||||
**Context**: Group captures by wardriving session for batch operations and statistics.
|
||||
|
||||
**Decision**: Add `session_id` to captures table (like Wigle's `run_id`).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's run_id groups **route points** by wardriving session
|
||||
- Enables batch operations (upload session, view session stats)
|
||||
- Useful for statistics (captures per session, coverage area)
|
||||
- Supports partial uploads (session-by-session)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- No grouping: Hard to track wardriving runs, no batch operations
|
||||
- Timestamp-based grouping: Unreliable (gaps in time)
|
||||
|
||||
**Implementation**:
|
||||
```sql
|
||||
CREATE TABLE sessions (
|
||||
session_id text primary key,
|
||||
user_id text not null,
|
||||
start_time bigint not null,
|
||||
end_time bigint,
|
||||
capture_count integer default 0,
|
||||
uploaded boolean default false
|
||||
);
|
||||
|
||||
ALTER TABLE captures ADD COLUMN session_id text REFERENCES sessions(session_id);
|
||||
CREATE INDEX idx_captures_session ON captures(session_id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 8: Performance - Transaction Batching
|
||||
|
||||
**Context**: Writing captures one-at-a-time is slow (disk I/O overhead).
|
||||
|
||||
**Decision**: Batch database writes in transactions (up to 512 operations).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle batches up to **512 DB updates** per transaction
|
||||
- Reduces disk I/O by 100x+ (one fsync per batch vs. per operation)
|
||||
- Critical for write-heavy workloads (wardriving generates 1000s of observations)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Individual transactions: 100x slower, disk thrashing
|
||||
- Larger batches (>1000): Increased memory, longer transaction locks
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
def batch_insert_captures(captures: List[Capture], batch_size=512):
|
||||
session = Session()
|
||||
|
||||
for i in range(0, len(captures), batch_size):
|
||||
batch = captures[i:i+batch_size]
|
||||
session.bulk_insert_mappings(Capture, batch)
|
||||
session.commit() # One commit per batch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 9: Caching - LRU Cache for Device Lookups
|
||||
|
||||
**Context**: Device signature matching queries database frequently for same devices.
|
||||
|
||||
**Decision**: Add LRU cache (64-256 entries) for device lookups.
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **64-entry LRU cache** for network lookups
|
||||
- Reduces database queries by 90%+ (most captures match known devices)
|
||||
- Small memory footprint (few KB per entry)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- No cache: Database bottleneck, slow matching
|
||||
- Redis cache: Overkill for simple lookups, adds network latency
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def get_device_by_id(device_id: int) -> Device:
|
||||
return db.query(Device).filter_by(device_id=device_id).first()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 10: API Authentication - JWT + API Keys
|
||||
|
||||
**Context**: Need to authenticate users for uploads and API access.
|
||||
|
||||
**Decision**: Support both JWT tokens (web) and API keys (scripts/tools).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **basic auth with API tokens** (retrieved via username/password)
|
||||
- JWT for web sessions (short-lived, stateless)
|
||||
- API keys for CLI tools (long-lived, revocable)
|
||||
- Flexible for different use cases
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Username/password only: Insecure for API, no token revocation
|
||||
- OAuth only: Overkill for simple use cases
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from fastapi import Depends, HTTPException
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
security = HTTPBearer()
|
||||
|
||||
def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
token = credentials.credentials
|
||||
|
||||
# Try JWT first
|
||||
try:
|
||||
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
|
||||
return payload["user_id"]
|
||||
except:
|
||||
pass
|
||||
|
||||
# Try API key
|
||||
user = db.query(User).filter_by(api_key=token).first()
|
||||
if user:
|
||||
return user.user_id
|
||||
|
||||
raise HTTPException(status_code=401, detail="Invalid authentication")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Decision | Wigle Pattern | GigLez Adaptation | Rationale |
|
||||
|----------|---------------|-------------------|-----------|
|
||||
| Database | SQLite 3-table | PostgreSQL + PostGIS 3-table | Geospatial queries, ACID |
|
||||
| Deduplication | BSSID primary key | SHA256 hash primary key | File-based, not MAC-based |
|
||||
| Upload Format | CSV | JSON + binary | Binary file support |
|
||||
| Processing | Background thread | Async task queue | Horizontal scaling |
|
||||
| GPS Validation | Multi-level checks | Same checks | Proven filtering |
|
||||
| Upload Marker | PREF_DB_MARKER | user_id → last_uploaded_id | Incremental sync |
|
||||
| Session Tracking | run_id | session_id | Batch operations |
|
||||
| Performance | Transaction batching | Same (512 ops) | Write optimization |
|
||||
| Caching | 64-entry LRU | 256-entry LRU | Larger device DB |
|
||||
| Auth | API tokens | JWT + API keys | Flexibility |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Implement PostgreSQL schema** (captures, devices, capture_matches, sessions)
|
||||
2. **Build .sub parser** with GPS validation
|
||||
3. **Create upload API** with background processing
|
||||
4. **Add device matching engine** (signature database)
|
||||
5. **Implement upload markers** for incremental sync
|
||||
6. **Add caching layer** (LRU for devices, Redis for API responses)
|
||||
7. **Build web interface** (map visualization, search, upload form)
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Last Updated**: 2026-01-12
|
||||
**References**: /docs/wigle_analysis.md
|
||||
@@ -0,0 +1,561 @@
|
||||
# Database Schema Design
|
||||
|
||||
## Overview
|
||||
|
||||
The GigLez database stores RF signal captures with GPS coordinates, device signatures, and community contributions. The schema supports efficient querying by location, frequency, protocol, and device type.
|
||||
|
||||
## Core Tables
|
||||
|
||||
### 1. captures
|
||||
|
||||
Primary table storing raw RF signal captures with GPS attribution.
|
||||
|
||||
```sql
|
||||
CREATE TABLE captures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
session_id INTEGER REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
|
||||
-- GPS Data
|
||||
latitude DECIMAL(10, 8) NOT NULL,
|
||||
longitude DECIMAL(11, 8) NOT NULL,
|
||||
altitude DECIMAL(8, 2),
|
||||
gps_accuracy DECIMAL(6, 2),
|
||||
|
||||
-- Timestamp
|
||||
captured_at TIMESTAMP NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- RF Signal Data
|
||||
frequency INTEGER NOT NULL, -- in Hz
|
||||
rssi INTEGER, -- Received Signal Strength Indicator
|
||||
modulation VARCHAR(50), -- OOK, 2FSK, etc.
|
||||
preset VARCHAR(100),
|
||||
|
||||
-- Protocol Information (if decoded)
|
||||
protocol VARCHAR(100),
|
||||
bit_length INTEGER,
|
||||
key_data BYTEA,
|
||||
timing_element INTEGER, -- TE value in microseconds
|
||||
|
||||
-- Raw Signal Data
|
||||
raw_data TEXT, -- RAW_Data timings
|
||||
raw_format VARCHAR(20), -- 'RAW', 'BinRAW', 'KEY'
|
||||
|
||||
-- File Storage
|
||||
file_path VARCHAR(500), -- Path to .sub file if stored separately
|
||||
file_hash VARCHAR(64), -- SHA-256 hash for deduplication
|
||||
|
||||
-- Matching
|
||||
device_id INTEGER REFERENCES devices(id),
|
||||
match_confidence DECIMAL(5, 4), -- 0.0 to 1.0
|
||||
match_method VARCHAR(50), -- 'auto', 'user', 'community'
|
||||
|
||||
-- Indexes for geospatial queries
|
||||
CONSTRAINT valid_latitude CHECK (latitude >= -90 AND latitude <= 90),
|
||||
CONSTRAINT valid_longitude CHECK (longitude >= -180 AND longitude <= 180)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_captures_location ON captures USING GIST (ll_to_earth(latitude, longitude));
|
||||
CREATE INDEX idx_captures_frequency ON captures(frequency);
|
||||
CREATE INDEX idx_captures_protocol ON captures(protocol);
|
||||
CREATE INDEX idx_captures_timestamp ON captures(captured_at DESC);
|
||||
CREATE INDEX idx_captures_session ON captures(session_id);
|
||||
CREATE INDEX idx_captures_device ON captures(device_id);
|
||||
CREATE INDEX idx_captures_hash ON captures(file_hash);
|
||||
```
|
||||
|
||||
### 2. sessions
|
||||
|
||||
Wardriving/capture sessions to group related captures.
|
||||
|
||||
```sql
|
||||
CREATE TABLE sessions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
user_id INTEGER REFERENCES users(id),
|
||||
|
||||
-- Session Metadata
|
||||
name VARCHAR(200),
|
||||
description TEXT,
|
||||
started_at TIMESTAMP NOT NULL DEFAULT NOW(),
|
||||
ended_at TIMESTAMP,
|
||||
|
||||
-- Privacy Settings
|
||||
is_public BOOLEAN DEFAULT true,
|
||||
anonymize_gps BOOLEAN DEFAULT false,
|
||||
gps_precision_meters INTEGER DEFAULT 10,
|
||||
|
||||
-- Session Statistics (computed)
|
||||
total_captures INTEGER DEFAULT 0,
|
||||
unique_devices INTEGER DEFAULT 0,
|
||||
distance_km DECIMAL(10, 2),
|
||||
|
||||
-- Bounding Box (for quick filtering)
|
||||
min_latitude DECIMAL(10, 8),
|
||||
max_latitude DECIMAL(10, 8),
|
||||
min_longitude DECIMAL(11, 8),
|
||||
max_longitude DECIMAL(11, 8)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sessions_user ON sessions(user_id);
|
||||
CREATE INDEX idx_sessions_started ON sessions(started_at DESC);
|
||||
```
|
||||
|
||||
### 3. devices
|
||||
|
||||
Known IoT device types with signature information.
|
||||
|
||||
```sql
|
||||
CREATE TABLE devices (
|
||||
id SERIAL PRIMARY KEY,
|
||||
|
||||
-- Device Identification
|
||||
manufacturer VARCHAR(200),
|
||||
model VARCHAR(200),
|
||||
device_type VARCHAR(100), -- 'garage_door', 'weather_station', 'key_fob', etc.
|
||||
description TEXT,
|
||||
|
||||
-- RF Characteristics
|
||||
typical_frequency INTEGER, -- Most common frequency in Hz
|
||||
frequency_range_low INTEGER,
|
||||
frequency_range_high INTEGER,
|
||||
modulation_types TEXT[], -- Array: ['OOK', '2FSK']
|
||||
|
||||
-- Protocol Information
|
||||
protocol VARCHAR(100),
|
||||
bit_length INTEGER,
|
||||
encoding VARCHAR(50), -- 'PWM', 'PPM', 'Manchester', etc.
|
||||
|
||||
-- Metadata
|
||||
fcc_id VARCHAR(50),
|
||||
manufacturer_code VARCHAR(50), -- For protocols like KeeLoq
|
||||
|
||||
-- Community Data
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
verified BOOLEAN DEFAULT false,
|
||||
verification_count INTEGER DEFAULT 0,
|
||||
|
||||
-- Source
|
||||
source VARCHAR(50), -- 'flipper', 'rtl433', 'urh', 'community'
|
||||
source_url TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX idx_devices_manufacturer ON devices(manufacturer);
|
||||
CREATE INDEX idx_devices_type ON devices(device_type);
|
||||
CREATE INDEX idx_devices_frequency ON devices(typical_frequency);
|
||||
CREATE INDEX idx_devices_protocol ON devices(protocol);
|
||||
```
|
||||
|
||||
### 4. signatures
|
||||
|
||||
Protocol signatures for device matching.
|
||||
|
||||
```sql
|
||||
CREATE TABLE signatures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
device_id INTEGER REFERENCES devices(id) ON DELETE CASCADE,
|
||||
|
||||
-- Signature Pattern
|
||||
protocol VARCHAR(100) NOT NULL,
|
||||
frequency INTEGER,
|
||||
modulation VARCHAR(50),
|
||||
|
||||
-- Matching Criteria
|
||||
bit_pattern BYTEA,
|
||||
bit_mask BYTEA, -- Which bits to match (1=match, 0=ignore)
|
||||
timing_min INTEGER, -- TE range in microseconds
|
||||
timing_max INTEGER,
|
||||
|
||||
-- Raw Pattern (for regex-like matching)
|
||||
pattern_regex TEXT,
|
||||
|
||||
-- Confidence Weighting
|
||||
weight DECIMAL(4, 3) DEFAULT 1.0, -- Higher = more reliable signature
|
||||
|
||||
-- Metadata
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
source VARCHAR(50),
|
||||
source_file VARCHAR(500), -- Original .sub or config file
|
||||
|
||||
UNIQUE(device_id, protocol, bit_pattern)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_signatures_device ON signatures(device_id);
|
||||
CREATE INDEX idx_signatures_protocol ON signatures(protocol);
|
||||
CREATE INDEX idx_signatures_frequency ON signatures(frequency);
|
||||
```
|
||||
|
||||
### 5. users
|
||||
|
||||
User accounts for community features.
|
||||
|
||||
```sql
|
||||
CREATE TABLE users (
|
||||
id SERIAL PRIMARY KEY,
|
||||
|
||||
-- Authentication
|
||||
username VARCHAR(50) UNIQUE NOT NULL,
|
||||
email VARCHAR(255) UNIQUE NOT NULL,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
|
||||
-- Profile
|
||||
display_name VARCHAR(100),
|
||||
avatar_url TEXT,
|
||||
bio TEXT,
|
||||
|
||||
-- Statistics
|
||||
total_captures INTEGER DEFAULT 0,
|
||||
total_identifications INTEGER DEFAULT 0,
|
||||
reputation_score INTEGER DEFAULT 0,
|
||||
|
||||
-- Settings
|
||||
api_key VARCHAR(64) UNIQUE,
|
||||
email_verified BOOLEAN DEFAULT false,
|
||||
|
||||
-- Timestamps
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
last_login TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_username ON users(username);
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
CREATE INDEX idx_users_api_key ON users(api_key);
|
||||
```
|
||||
|
||||
### 6. identifications
|
||||
|
||||
User-submitted device identifications (with photos).
|
||||
|
||||
```sql
|
||||
CREATE TABLE identifications (
|
||||
id SERIAL PRIMARY KEY,
|
||||
capture_id INTEGER REFERENCES captures(id) ON DELETE CASCADE,
|
||||
device_id INTEGER REFERENCES devices(id),
|
||||
user_id INTEGER REFERENCES users(id),
|
||||
|
||||
-- Identification Details
|
||||
confidence VARCHAR(20), -- 'certain', 'likely', 'guess'
|
||||
notes TEXT,
|
||||
|
||||
-- Visual Evidence
|
||||
photo_urls TEXT[], -- Array of image URLs
|
||||
|
||||
-- Community Validation
|
||||
upvotes INTEGER DEFAULT 0,
|
||||
downvotes INTEGER DEFAULT 0,
|
||||
verified BOOLEAN DEFAULT false,
|
||||
verified_by INTEGER REFERENCES users(id),
|
||||
verified_at TIMESTAMP,
|
||||
|
||||
-- Timestamps
|
||||
submitted_at TIMESTAMP DEFAULT NOW(),
|
||||
|
||||
UNIQUE(capture_id, user_id, device_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_identifications_capture ON identifications(capture_id);
|
||||
CREATE INDEX idx_identifications_device ON identifications(device_id);
|
||||
CREATE INDEX idx_identifications_user ON identifications(user_id);
|
||||
```
|
||||
|
||||
### 7. votes
|
||||
|
||||
Voting on device identifications.
|
||||
|
||||
```sql
|
||||
CREATE TABLE votes (
|
||||
id SERIAL PRIMARY KEY,
|
||||
identification_id INTEGER REFERENCES identifications(id) ON DELETE CASCADE,
|
||||
user_id INTEGER REFERENCES users(id),
|
||||
|
||||
vote_type INTEGER NOT NULL, -- 1 = upvote, -1 = downvote
|
||||
voted_at TIMESTAMP DEFAULT NOW(),
|
||||
|
||||
UNIQUE(identification_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_votes_identification ON votes(identification_id);
|
||||
CREATE INDEX idx_votes_user ON votes(user_id);
|
||||
```
|
||||
|
||||
### 8. flipper_signatures
|
||||
|
||||
Imported Flipper Zero .sub file signatures.
|
||||
|
||||
```sql
|
||||
CREATE TABLE flipper_signatures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
signature_id INTEGER REFERENCES signatures(id) ON DELETE CASCADE,
|
||||
|
||||
-- Flipper-Specific Fields
|
||||
filetype VARCHAR(50), -- 'Flipper SubGhz Key File' or 'RAW File'
|
||||
version INTEGER,
|
||||
preset VARCHAR(100),
|
||||
|
||||
-- Custom Preset Data
|
||||
custom_preset_module VARCHAR(50),
|
||||
custom_preset_data BYTEA,
|
||||
|
||||
-- Protocol Data
|
||||
protocol VARCHAR(100),
|
||||
bit INTEGER,
|
||||
key BYTEA,
|
||||
te INTEGER, -- Timing element
|
||||
|
||||
-- RAW Data
|
||||
raw_data TEXT,
|
||||
bin_raw_bit INTEGER,
|
||||
bin_raw_te INTEGER,
|
||||
bin_raw_data BYTEA,
|
||||
|
||||
-- Source
|
||||
source_file VARCHAR(500),
|
||||
imported_at TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_flipper_protocol ON flipper_signatures(protocol);
|
||||
CREATE INDEX idx_flipper_preset ON flipper_signatures(preset);
|
||||
```
|
||||
|
||||
### 9. rtl433_protocols
|
||||
|
||||
Imported RTL_433 protocol definitions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE rtl433_protocols (
|
||||
id SERIAL PRIMARY KEY,
|
||||
signature_id INTEGER REFERENCES signatures(id) ON DELETE CASCADE,
|
||||
|
||||
-- Protocol Identification
|
||||
protocol_number INTEGER,
|
||||
protocol_name VARCHAR(200),
|
||||
model VARCHAR(200),
|
||||
|
||||
-- RF Characteristics
|
||||
frequency INTEGER,
|
||||
modulation VARCHAR(50), -- 'OOK_PWM', 'FSK_PCM', etc.
|
||||
|
||||
-- Timing Information (in microseconds)
|
||||
short_width INTEGER,
|
||||
long_width INTEGER,
|
||||
reset_limit INTEGER,
|
||||
gap_limit INTEGER,
|
||||
|
||||
-- Decoding
|
||||
decoder_type VARCHAR(50),
|
||||
bit_count INTEGER,
|
||||
|
||||
-- JSON Fields Mapping
|
||||
json_fields JSONB, -- Expected output fields
|
||||
|
||||
-- Source
|
||||
source_file VARCHAR(500),
|
||||
imported_at TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_rtl433_protocol_num ON rtl433_protocols(protocol_number);
|
||||
CREATE INDEX idx_rtl433_model ON rtl433_protocols(model);
|
||||
CREATE INDEX idx_rtl433_modulation ON rtl433_protocols(modulation);
|
||||
```
|
||||
|
||||
## Materialized Views
|
||||
|
||||
### device_statistics
|
||||
|
||||
Pre-computed statistics for device types.
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW device_statistics AS
|
||||
SELECT
|
||||
d.id as device_id,
|
||||
d.manufacturer,
|
||||
d.model,
|
||||
COUNT(c.id) as total_captures,
|
||||
COUNT(DISTINCT c.session_id) as total_sessions,
|
||||
MIN(c.captured_at) as first_seen,
|
||||
MAX(c.captured_at) as last_seen,
|
||||
AVG(c.rssi) as avg_rssi,
|
||||
ST_Collect(ST_MakePoint(c.longitude, c.latitude)) as capture_locations
|
||||
FROM devices d
|
||||
LEFT JOIN captures c ON c.device_id = d.id
|
||||
GROUP BY d.id, d.manufacturer, d.model;
|
||||
|
||||
CREATE INDEX idx_device_stats_device ON device_statistics(device_id);
|
||||
```
|
||||
|
||||
### geographic_heatmap
|
||||
|
||||
Aggregated capture density for mapping.
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW geographic_heatmap AS
|
||||
SELECT
|
||||
ROUND(latitude::numeric, 3) as lat_bucket,
|
||||
ROUND(longitude::numeric, 3) as lon_bucket,
|
||||
COUNT(*) as capture_count,
|
||||
COUNT(DISTINCT device_id) as unique_devices,
|
||||
array_agg(DISTINCT protocol) as protocols_seen
|
||||
FROM captures
|
||||
WHERE device_id IS NOT NULL
|
||||
GROUP BY lat_bucket, lon_bucket;
|
||||
|
||||
CREATE INDEX idx_heatmap_location ON geographic_heatmap(lat_bucket, lon_bucket);
|
||||
```
|
||||
|
||||
## Functions
|
||||
|
||||
### calculate_distance
|
||||
|
||||
Calculate distance between two GPS coordinates.
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION calculate_distance(
|
||||
lat1 DECIMAL, lon1 DECIMAL,
|
||||
lat2 DECIMAL, lon2 DECIMAL
|
||||
) RETURNS DECIMAL AS $$
|
||||
DECLARE
|
||||
R DECIMAL := 6371.0; -- Earth radius in km
|
||||
dLat DECIMAL;
|
||||
dLon DECIMAL;
|
||||
a DECIMAL;
|
||||
c DECIMAL;
|
||||
BEGIN
|
||||
dLat := radians(lat2 - lat1);
|
||||
dLon := radians(lon2 - lon1);
|
||||
a := sin(dLat/2) * sin(dLat/2) +
|
||||
cos(radians(lat1)) * cos(radians(lat2)) *
|
||||
sin(dLon/2) * sin(dLon/2);
|
||||
c := 2 * atan2(sqrt(a), sqrt(1-a));
|
||||
RETURN R * c;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql IMMUTABLE;
|
||||
```
|
||||
|
||||
### match_signature
|
||||
|
||||
Match a capture against all known signatures.
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION match_signature(
|
||||
p_capture_id INTEGER
|
||||
) RETURNS TABLE(device_id INTEGER, confidence DECIMAL) AS $$
|
||||
BEGIN
|
||||
RETURN QUERY
|
||||
SELECT
|
||||
s.device_id,
|
||||
CASE
|
||||
WHEN c.protocol = s.protocol
|
||||
AND c.frequency = s.frequency
|
||||
AND c.bit_length = s.bit_length THEN 1.0
|
||||
WHEN c.protocol = s.protocol
|
||||
AND c.frequency = s.frequency THEN 0.8
|
||||
WHEN c.protocol = s.protocol THEN 0.5
|
||||
ELSE 0.0
|
||||
END as confidence
|
||||
FROM captures c
|
||||
CROSS JOIN signatures s
|
||||
WHERE c.id = p_capture_id
|
||||
AND c.protocol IS NOT NULL
|
||||
ORDER BY confidence DESC
|
||||
LIMIT 10;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
```
|
||||
|
||||
## Triggers
|
||||
|
||||
### update_session_statistics
|
||||
|
||||
Automatically update session statistics when captures are added.
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION update_session_stats()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
UPDATE sessions SET
|
||||
total_captures = (SELECT COUNT(*) FROM captures WHERE session_id = NEW.session_id),
|
||||
unique_devices = (SELECT COUNT(DISTINCT device_id) FROM captures WHERE session_id = NEW.session_id),
|
||||
min_latitude = LEAST(min_latitude, NEW.latitude),
|
||||
max_latitude = GREATEST(max_latitude, NEW.latitude),
|
||||
min_longitude = LEAST(min_longitude, NEW.longitude),
|
||||
max_longitude = GREATEST(max_longitude, NEW.longitude)
|
||||
WHERE id = NEW.session_id;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER trigger_update_session_stats
|
||||
AFTER INSERT ON captures
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_session_stats();
|
||||
```
|
||||
|
||||
### update_device_verification
|
||||
|
||||
Update device verification status based on identification votes.
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION update_device_verification()
|
||||
RETURNS TRIGGER AS $$
|
||||
DECLARE
|
||||
net_votes INTEGER;
|
||||
BEGIN
|
||||
SELECT (upvotes - downvotes) INTO net_votes
|
||||
FROM identifications
|
||||
WHERE id = NEW.identification_id;
|
||||
|
||||
IF net_votes >= 5 THEN
|
||||
UPDATE identifications SET verified = true
|
||||
WHERE id = NEW.identification_id;
|
||||
END IF;
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER trigger_update_verification
|
||||
AFTER INSERT OR UPDATE ON votes
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_device_verification();
|
||||
```
|
||||
|
||||
## Sample Queries
|
||||
|
||||
### Find all captures near a location
|
||||
|
||||
```sql
|
||||
SELECT c.*, d.manufacturer, d.model
|
||||
FROM captures c
|
||||
LEFT JOIN devices d ON c.device_id = d.id
|
||||
WHERE calculate_distance(c.latitude, c.longitude, 40.7128, -74.0060) <= 1.0 -- Within 1km
|
||||
ORDER BY c.captured_at DESC;
|
||||
```
|
||||
|
||||
### Get device density heatmap
|
||||
|
||||
```sql
|
||||
SELECT lat_bucket, lon_bucket, capture_count, unique_devices
|
||||
FROM geographic_heatmap
|
||||
WHERE capture_count > 5
|
||||
ORDER BY capture_count DESC;
|
||||
```
|
||||
|
||||
### Find unidentified captures
|
||||
|
||||
```sql
|
||||
SELECT c.id, c.frequency, c.protocol, c.latitude, c.longitude, c.captured_at
|
||||
FROM captures c
|
||||
WHERE c.device_id IS NULL
|
||||
AND c.protocol IS NOT NULL
|
||||
ORDER BY c.captured_at DESC
|
||||
LIMIT 100;
|
||||
```
|
||||
|
||||
### Top device types by capture count
|
||||
|
||||
```sql
|
||||
SELECT d.manufacturer, d.model, d.device_type, COUNT(c.id) as captures
|
||||
FROM devices d
|
||||
JOIN captures c ON c.device_id = d.id
|
||||
GROUP BY d.id, d.manufacturer, d.model, d.device_type
|
||||
ORDER BY captures DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
@@ -0,0 +1,569 @@
|
||||
# Signature Database Integration
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes how to integrate and use signature databases from Flipper Zero, RTL_433, and community sources for device identification.
|
||||
|
||||
## 1. Flipper Zero Sub-GHz Database
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/flipperdevices/flipperzero-firmware
|
||||
- **Location**: `/assets/resources/subghz/assets/` in firmware repo
|
||||
- **Format**: `.sub` files
|
||||
|
||||
### File Format Structure
|
||||
|
||||
#### Standard Key File
|
||||
```
|
||||
Filetype: Flipper SubGhz Key File
|
||||
Version: 1
|
||||
Frequency: 433920000
|
||||
Preset: FuriHalSubGhzPresetOok270Async
|
||||
Protocol: Princeton
|
||||
Bit: 24
|
||||
Key: 00 00 00 00 00 95 D5 D4
|
||||
TE: 400
|
||||
```
|
||||
|
||||
**Field Descriptions:**
|
||||
- `Filetype`: Must be "Flipper SubGhz Key File" for protocol files
|
||||
- `Version`: File format version (currently 1)
|
||||
- `Frequency`: Operating frequency in Hz (e.g., 433920000 = 433.92 MHz)
|
||||
- `Preset`: Modulation preset (see Preset Types below)
|
||||
- `Protocol`: Protocol name (Princeton, KeeLoq, Star Line, etc.)
|
||||
- `Bit`: Number of bits in the transmission
|
||||
- `Key`: Hex-encoded data payload
|
||||
- `TE`: Timing element in microseconds (pulse width)
|
||||
|
||||
#### RAW Signal File
|
||||
```
|
||||
Filetype: Flipper SubGhz RAW File
|
||||
Version: 1
|
||||
Frequency: 433920000
|
||||
Preset: FuriHalSubGhzPresetOok650Async
|
||||
Protocol: RAW
|
||||
RAW_Data: 29262 361 -68 2635 -66 24113 -66 11 -66 11 -132
|
||||
```
|
||||
|
||||
**RAW_Data Format:**
|
||||
- Array of timing values in microseconds
|
||||
- Positive values = carrier ON
|
||||
- Negative values = carrier OFF
|
||||
- Must alternate between positive and negative
|
||||
- Up to 512 values per line
|
||||
|
||||
#### BinRAW File (Compressed)
|
||||
```
|
||||
Filetype: Flipper SubGhz RAW File
|
||||
Version: 1
|
||||
Frequency: 315000000
|
||||
Preset: FuriHalSubGhzPreset2FSKDev238Async
|
||||
Protocol: BinRAW
|
||||
Bit: 1572
|
||||
TE: 597
|
||||
Bit_RAW: 260
|
||||
Data_RAW: 00 00 00 00 AA AA AA AA 0F 4A B5 55
|
||||
```
|
||||
|
||||
**BinRAW Format:**
|
||||
- `Bit`: Total bits in transmission
|
||||
- `TE`: Timing element (microsecond per bit)
|
||||
- `Bit_RAW`: Number of bits in compressed format
|
||||
- `Data_RAW`: Bit-packed data (1=carrier, 0=gap)
|
||||
|
||||
### Preset Types
|
||||
|
||||
| Preset | Modulation | Bandwidth | Deviation | Use Case |
|
||||
|--------|------------|-----------|-----------|----------|
|
||||
| `FuriHalSubGhzPresetOok270Async` | OOK | 270 kHz | - | Standard garage doors, remotes |
|
||||
| `FuriHalSubGhzPresetOok650Async` | OOK | 650 kHz | - | Fast protocols, doorbells |
|
||||
| `FuriHalSubGhzPreset2FSKDev238Async` | 2FSK | 270 kHz | 2.38 kHz | TPMS, some sensors |
|
||||
| `FuriHalSubGhzPreset2FSKDev476Async` | 2FSK | 270 kHz | 47.6 kHz | High-deviation FSK |
|
||||
| `FuriHalSubGhzPresetCustom` | Custom | Varies | Varies | User-defined configs |
|
||||
|
||||
### Custom Presets
|
||||
|
||||
Custom presets allow specific CC1101 register configurations:
|
||||
|
||||
```
|
||||
Filetype: Flipper SubGhz Key File
|
||||
Version: 1
|
||||
Frequency: 868350000
|
||||
Preset: FuriHalSubGhzPresetCustom
|
||||
Custom_preset_module: CC1101
|
||||
Custom_preset_data: 02 0D 03 07 08 32 0B 06 14 00 13 00 12 00 11 32 10 17 18 18 19 18 1D 91 1C 00 1B 07 20 FB 22 11 21 B6 00 00 C0 00 00 00 00 00 00 00
|
||||
Protocol: StarLine
|
||||
...
|
||||
```
|
||||
|
||||
### Common Protocols
|
||||
|
||||
| Protocol | Frequency | Modulation | Bit Length | Use Case |
|
||||
|----------|-----------|------------|------------|----------|
|
||||
| Princeton | 433.92 MHz | OOK_PWM | 24 | Generic remotes, garage doors |
|
||||
| KeeLoq | 433.92 MHz | OOK | 64-66 | Car key fobs, secure remotes |
|
||||
| Star Line | 433.92 MHz | OOK | Varies | Car alarm systems |
|
||||
| Came | 433.92 MHz | OOK | 12 | Gate openers |
|
||||
| Nice FLO | 433.92 MHz | OOK | 12-24 | Gate openers |
|
||||
| Somfy Telis | 433.42 MHz | OOK | 56 | Window blinds |
|
||||
| Holtek HT12X | 433.92 MHz | OOK_PWM | 12 | Generic remotes |
|
||||
|
||||
### Importing Flipper Signatures
|
||||
|
||||
#### Step 1: Clone the Repository
|
||||
```bash
|
||||
cd signatures/flipper
|
||||
git clone --depth 1 https://github.com/flipperdevices/flipperzero-firmware.git temp
|
||||
cp -r temp/assets/resources/subghz/assets/* ./
|
||||
rm -rf temp
|
||||
```
|
||||
|
||||
#### Step 2: Parse .sub Files
|
||||
```python
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def parse_flipper_sub(file_path):
|
||||
"""Parse a Flipper Zero .sub file"""
|
||||
data = {}
|
||||
with open(file_path, 'r') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if ':' in line:
|
||||
key, value = line.split(':', 1)
|
||||
data[key.strip()] = value.strip()
|
||||
return data
|
||||
|
||||
# Example usage
|
||||
sub_file = Path('signatures/flipper/princeton_433.sub')
|
||||
parsed = parse_flipper_sub(sub_file)
|
||||
print(f"Protocol: {parsed.get('Protocol')}")
|
||||
print(f"Frequency: {parsed.get('Frequency')} Hz")
|
||||
print(f"Key: {parsed.get('Key')}")
|
||||
```
|
||||
|
||||
#### Step 3: Import to Database
|
||||
```python
|
||||
def import_flipper_signature(parsed_data, device_id):
|
||||
"""Import parsed .sub file to database"""
|
||||
signature = {
|
||||
'device_id': device_id,
|
||||
'protocol': parsed_data.get('Protocol'),
|
||||
'frequency': int(parsed_data.get('Frequency', 0)),
|
||||
'modulation': parse_preset_modulation(parsed_data.get('Preset')),
|
||||
'bit_pattern': bytes.fromhex(parsed_data.get('Key', '').replace(' ', '')),
|
||||
'timing_min': int(parsed_data.get('TE', 0)) * 0.9, # 10% tolerance
|
||||
'timing_max': int(parsed_data.get('TE', 0)) * 1.1,
|
||||
'source': 'flipper',
|
||||
'source_file': str(file_path)
|
||||
}
|
||||
# Insert into database
|
||||
db.signatures.insert(signature)
|
||||
```
|
||||
|
||||
## 2. RTL_433 Protocol Database
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/merbanan/rtl_433
|
||||
- **Location**: `/src/devices/*.c` (protocol implementations)
|
||||
- **Documentation**: https://triq.org/rtl_433/
|
||||
|
||||
### Protocol Structure
|
||||
|
||||
RTL_433 protocols are defined in C code with decoder specifications:
|
||||
|
||||
```c
|
||||
static char* output_fields[] = {
|
||||
"model",
|
||||
"id",
|
||||
"channel",
|
||||
"battery_ok",
|
||||
"temperature_C",
|
||||
"humidity",
|
||||
"mic",
|
||||
NULL,
|
||||
};
|
||||
|
||||
r_device acurite_tower = {
|
||||
.name = "Acurite-Tower",
|
||||
.modulation = OOK_PULSE_PWM,
|
||||
.short_width = 220,
|
||||
.long_width = 440,
|
||||
.reset_limit = 900,
|
||||
.decode_fn = &acurite_tower_decode,
|
||||
.fields = output_fields,
|
||||
};
|
||||
```
|
||||
|
||||
### JSON Output Format
|
||||
|
||||
RTL_433 outputs decoded data as JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"time": "2025-01-11 20:15:32",
|
||||
"model": "Acurite-Tower",
|
||||
"id": 12345,
|
||||
"channel": "A",
|
||||
"battery_ok": 1,
|
||||
"temperature_C": 22.5,
|
||||
"humidity": 45,
|
||||
"mic": "CRC"
|
||||
}
|
||||
```
|
||||
|
||||
### Common Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `time` | String | Timestamp in ISO 8601 format |
|
||||
| `model` | String | Manufacturer-Model identifier |
|
||||
| `id` | Integer | Unique device ID |
|
||||
| `channel` | String | Channel identifier (A, B, C, etc.) |
|
||||
| `battery_ok` | Integer | Battery status (0=low, 1=ok) |
|
||||
| `temperature_C` | Float | Temperature in Celsius |
|
||||
| `humidity` | Integer | Relative humidity percentage |
|
||||
| `mic` | String | Message integrity check type |
|
||||
|
||||
### Modulation Types
|
||||
|
||||
| Modulation | Description | Common Devices |
|
||||
|------------|-------------|----------------|
|
||||
| `OOK_PWM` | Pulse Width Modulation | Remotes, sensors |
|
||||
| `OOK_PPM` | Pulse Position Modulation | Weather stations |
|
||||
| `OOK_PCM` | Pulse Code Modulation | Security sensors |
|
||||
| `FSK_PCM` | FSK Pulse Code | TPMS, smart meters |
|
||||
| `FSK_PWM` | FSK Pulse Width | Advanced sensors |
|
||||
|
||||
### Extracting Protocol Definitions
|
||||
|
||||
#### Step 1: Extract from Source Code
|
||||
```bash
|
||||
cd signatures/rtl433
|
||||
git clone --depth 1 https://github.com/merbanan/rtl_433.git temp
|
||||
grep -r "r_device" temp/src/devices/*.c > protocols.txt
|
||||
rm -rf temp
|
||||
```
|
||||
|
||||
#### Step 2: Parse Protocol Definitions
|
||||
```python
|
||||
import re
|
||||
import json
|
||||
|
||||
def parse_rtl433_protocol(protocol_string):
|
||||
"""Extract protocol definition from C code"""
|
||||
pattern = r'r_device\s+(\w+)\s*=\s*{([^}]+)}'
|
||||
match = re.search(pattern, protocol_string, re.DOTALL)
|
||||
|
||||
if not match:
|
||||
return None
|
||||
|
||||
name = match.group(1)
|
||||
body = match.group(2)
|
||||
|
||||
# Extract fields
|
||||
modulation = re.search(r'\.modulation\s*=\s*(\w+)', body)
|
||||
short_width = re.search(r'\.short_width\s*=\s*(\d+)', body)
|
||||
long_width = re.search(r'\.long_width\s*=\s*(\d+)', body)
|
||||
model_name = re.search(r'\.name\s*=\s*"([^"]+)"', body)
|
||||
|
||||
return {
|
||||
'protocol_name': name,
|
||||
'model': model_name.group(1) if model_name else None,
|
||||
'modulation': modulation.group(1) if modulation else None,
|
||||
'short_width': int(short_width.group(1)) if short_width else None,
|
||||
'long_width': int(long_width.group(1)) if long_width else None
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Generate Protocol Database
|
||||
```python
|
||||
# Read all protocol definitions
|
||||
protocols = []
|
||||
for c_file in Path('temp/src/devices').glob('*.c'):
|
||||
with open(c_file, 'r') as f:
|
||||
content = f.read()
|
||||
parsed = parse_rtl433_protocol(content)
|
||||
if parsed:
|
||||
protocols.append(parsed)
|
||||
|
||||
# Save as JSON
|
||||
with open('signatures/rtl433/protocols.json', 'w') as f:
|
||||
json.dump(protocols, f, indent=2)
|
||||
```
|
||||
|
||||
### Using rtl_433 Test Data
|
||||
|
||||
RTL_433 provides test signals with expected JSON output:
|
||||
|
||||
```bash
|
||||
# Clone test repository
|
||||
git clone https://github.com/merbanan/rtl_433_tests.git signatures/rtl433/tests
|
||||
|
||||
# Test files are organized as:
|
||||
# rtl_433_tests/tests/{protocol_name}/{signal_type}/*.cu8
|
||||
# rtl_433_tests/tests/{protocol_name}/{signal_type}/*.json
|
||||
```
|
||||
|
||||
Example test data structure:
|
||||
```
|
||||
rtl_433_tests/tests/acurite/01/
|
||||
├── g001_433.92M_250k.cu8 # Raw signal capture
|
||||
└── g001_433.92M_250k.json # Expected decoded output
|
||||
```
|
||||
|
||||
### Importing RTL_433 Signatures
|
||||
|
||||
```python
|
||||
def import_rtl433_protocol(protocol_data, device_id):
|
||||
"""Import RTL_433 protocol to database"""
|
||||
# Determine frequency from modulation
|
||||
frequency_map = {
|
||||
'OOK': 433920000,
|
||||
'FSK': 868000000 # Varies by device
|
||||
}
|
||||
|
||||
modulation_type = protocol_data.get('modulation', 'OOK')
|
||||
base_freq = frequency_map.get(modulation_type[:3], 433920000)
|
||||
|
||||
signature = {
|
||||
'device_id': device_id,
|
||||
'protocol': protocol_data.get('model'),
|
||||
'frequency': base_freq,
|
||||
'modulation': modulation_type,
|
||||
'timing_min': protocol_data.get('short_width'),
|
||||
'timing_max': protocol_data.get('long_width'),
|
||||
'source': 'rtl433',
|
||||
'source_file': protocol_data.get('source_file')
|
||||
}
|
||||
|
||||
# Insert into rtl433_protocols table
|
||||
db.rtl433_protocols.insert({
|
||||
'protocol_name': protocol_data.get('protocol_name'),
|
||||
'model': protocol_data.get('model'),
|
||||
'modulation': modulation_type,
|
||||
'short_width': protocol_data.get('short_width'),
|
||||
'long_width': protocol_data.get('long_width'),
|
||||
'json_fields': protocol_data.get('fields', {})
|
||||
})
|
||||
|
||||
# Also create general signature
|
||||
db.signatures.insert(signature)
|
||||
```
|
||||
|
||||
## 3. Universal Radio Hacker (URH)
|
||||
|
||||
### Source
|
||||
- **Repository**: https://github.com/jopohl/urh
|
||||
- **Format**: `.urh` project files, `.complex` signal files
|
||||
|
||||
### Signal File Formats
|
||||
|
||||
#### .complex Files
|
||||
Raw I/Q signal data in binary format:
|
||||
- Interleaved I and Q samples
|
||||
- Typically 32-bit float or 8-bit integer
|
||||
- Sample rate metadata in accompanying .cfl file
|
||||
|
||||
#### .urh Project Files
|
||||
XML-based project containing:
|
||||
- Signal definitions
|
||||
- Demodulation parameters
|
||||
- Protocol structure
|
||||
- Labeled message fields
|
||||
|
||||
### Extracting URH Protocols
|
||||
|
||||
URH projects often contain valuable protocol information in the community wiki.
|
||||
|
||||
```bash
|
||||
# Download community-shared URH projects
|
||||
cd signatures/urh
|
||||
# Check URH GitHub wiki for shared project links
|
||||
```
|
||||
|
||||
## 4. Community Signatures
|
||||
|
||||
### User Submission Format
|
||||
|
||||
When users identify a device, they can submit:
|
||||
|
||||
```json
|
||||
{
|
||||
"device": {
|
||||
"manufacturer": "Chamberlain",
|
||||
"model": "KLIK3U-SS",
|
||||
"type": "garage_door_opener",
|
||||
"fcc_id": "K49KLIK3U"
|
||||
},
|
||||
"signature": {
|
||||
"frequency": 433920000,
|
||||
"protocol": "Security+ 2.0",
|
||||
"modulation": "OOK",
|
||||
"notes": "Rolling code, 3-button remote"
|
||||
},
|
||||
"evidence": {
|
||||
"photos": ["device_front.jpg", "device_back.jpg", "fcc_label.jpg"],
|
||||
"capture_file": "chamberlain_klik3u_001.sub"
|
||||
},
|
||||
"location": {
|
||||
"latitude": 40.7128,
|
||||
"longitude": -74.0060,
|
||||
"accuracy": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Verification Process
|
||||
|
||||
1. User submits identification with photo evidence
|
||||
2. Community members vote (upvote/downvote)
|
||||
3. After 5 net upvotes, identification is auto-verified
|
||||
4. Verified identifications create new signature entries
|
||||
5. High-reputation users can verify immediately
|
||||
|
||||
## 5. Signature Matching Algorithm
|
||||
|
||||
### Matching Strategy
|
||||
|
||||
```python
|
||||
def match_capture_to_signatures(capture):
|
||||
"""
|
||||
Match a capture against known signatures
|
||||
Returns list of (device_id, confidence) tuples
|
||||
"""
|
||||
matches = []
|
||||
|
||||
# Exact protocol + frequency + timing match
|
||||
exact = db.query('''
|
||||
SELECT device_id, 1.0 as confidence
|
||||
FROM signatures
|
||||
WHERE protocol = ? AND frequency = ? AND ? BETWEEN timing_min AND timing_max
|
||||
''', [capture.protocol, capture.frequency, capture.timing_element])
|
||||
matches.extend(exact)
|
||||
|
||||
# Protocol + frequency match (80% confidence)
|
||||
protocol_freq = db.query('''
|
||||
SELECT device_id, 0.8 as confidence
|
||||
FROM signatures
|
||||
WHERE protocol = ? AND frequency = ?
|
||||
''', [capture.protocol, capture.frequency])
|
||||
matches.extend(protocol_freq)
|
||||
|
||||
# Bit pattern matching (if available)
|
||||
if capture.key_data:
|
||||
pattern_matches = match_bit_pattern(capture.key_data)
|
||||
matches.extend(pattern_matches)
|
||||
|
||||
# De-duplicate and sort by confidence
|
||||
unique_matches = {}
|
||||
for device_id, conf in matches:
|
||||
if device_id not in unique_matches or conf > unique_matches[device_id]:
|
||||
unique_matches[device_id] = conf
|
||||
|
||||
return sorted(unique_matches.items(), key=lambda x: x[1], reverse=True)
|
||||
```
|
||||
|
||||
### Bit Pattern Matching
|
||||
|
||||
```python
|
||||
def match_bit_pattern(key_data, signatures):
|
||||
"""
|
||||
Match key data against signature patterns with masks
|
||||
"""
|
||||
matches = []
|
||||
|
||||
for sig in signatures:
|
||||
if not sig.bit_mask:
|
||||
continue
|
||||
|
||||
# Apply mask and compare
|
||||
masked_capture = apply_mask(key_data, sig.bit_mask)
|
||||
masked_signature = apply_mask(sig.bit_pattern, sig.bit_mask)
|
||||
|
||||
if masked_capture == masked_signature:
|
||||
confidence = 0.9 * sig.weight
|
||||
matches.append((sig.device_id, confidence))
|
||||
|
||||
return matches
|
||||
|
||||
def apply_mask(data, mask):
|
||||
"""Bitwise AND operation on byte arrays"""
|
||||
return bytes(a & b for a, b in zip(data, mask))
|
||||
```
|
||||
|
||||
## 6. Database Import Scripts
|
||||
|
||||
### Complete Import Pipeline
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Import all signature databases"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from importers import flipper, rtl433, urh
|
||||
|
||||
def main():
|
||||
print("Starting signature database import...")
|
||||
|
||||
# Import Flipper Zero signatures
|
||||
print("\n[1/3] Importing Flipper Zero .sub files...")
|
||||
flipper_dir = Path('signatures/flipper')
|
||||
flipper_count = flipper.import_all(flipper_dir)
|
||||
print(f" Imported {flipper_count} Flipper signatures")
|
||||
|
||||
# Import RTL_433 protocols
|
||||
print("\n[2/3] Importing RTL_433 protocols...")
|
||||
rtl433_file = Path('signatures/rtl433/protocols.json')
|
||||
rtl433_count = rtl433.import_protocols(rtl433_file)
|
||||
print(f" Imported {rtl433_count} RTL_433 protocols")
|
||||
|
||||
# Import URH community signals
|
||||
print("\n[3/3] Importing URH signals...")
|
||||
urh_dir = Path('signatures/urh')
|
||||
urh_count = urh.import_all(urh_dir)
|
||||
print(f" Imported {urh_count} URH signatures")
|
||||
|
||||
print(f"\nTotal signatures imported: {flipper_count + rtl433_count + urh_count}")
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
```
|
||||
|
||||
## 7. Signature Database Maintenance
|
||||
|
||||
### Regular Updates
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/update_signatures.sh
|
||||
|
||||
cd signatures/flipper
|
||||
git pull
|
||||
|
||||
cd ../rtl433
|
||||
git pull
|
||||
|
||||
# Re-import updated signatures
|
||||
python3 scripts/import_signatures.py --update
|
||||
```
|
||||
|
||||
### Quality Metrics
|
||||
|
||||
Track signature effectiveness:
|
||||
|
||||
```sql
|
||||
-- Signature match success rate
|
||||
SELECT
|
||||
s.id,
|
||||
d.manufacturer,
|
||||
d.model,
|
||||
COUNT(c.id) as total_matches,
|
||||
AVG(c.match_confidence) as avg_confidence
|
||||
FROM signatures s
|
||||
JOIN devices d ON s.device_id = d.id
|
||||
LEFT JOIN captures c ON c.device_id = d.id AND c.match_method = 'auto'
|
||||
GROUP BY s.id, d.manufacturer, d.model
|
||||
ORDER BY total_matches DESC;
|
||||
```
|
||||
@@ -0,0 +1,662 @@
|
||||
# LilyGo T-Embed Setup and Communication Protocol
|
||||
|
||||
## Overview
|
||||
|
||||
The LilyGo T-Embed is an ESP32-based device with an integrated CC1101 sub-GHz transceiver. This document covers hardware setup, firmware options, and the communication protocol between Android Termux and the T-Embed.
|
||||
|
||||
## Hardware Specifications
|
||||
|
||||
### LilyGo T-Embed CC1101
|
||||
- **MCU**: ESP32-S3 (Dual-core 240MHz)
|
||||
- **Display**: 1.9" ST7789 TFT (170x320 pixels)
|
||||
- **RF Chip**: CC1101 Sub-GHz transceiver
|
||||
- **Frequency Range**:
|
||||
- 300-348 MHz
|
||||
- 387-464 MHz
|
||||
- 779-928 MHz
|
||||
- **Storage**: MicroSD card slot
|
||||
- **Connectivity**: USB-C (serial + power)
|
||||
- **Battery**: Built-in LiPo charger
|
||||
|
||||
### CC1101 Transceiver Capabilities
|
||||
- **Modulation**: ASK/OOK, FSK, GFSK, MSK
|
||||
- **Data Rate**: 0.6 - 500 kbps
|
||||
- **RX Sensitivity**: -112 dBm @ 1.2 kbps
|
||||
- **TX Power**: Configurable up to +12 dBm
|
||||
- **FIFO Buffer**: 64 bytes RX/TX
|
||||
|
||||
## Firmware Options
|
||||
|
||||
### 1. Bruce Firmware (Recommended)
|
||||
**Repository**: https://github.com/pr3y/Bruce
|
||||
|
||||
Bruce is a multi-tool firmware with excellent Sub-GHz support designed for T-Embed.
|
||||
|
||||
**Features**:
|
||||
- Read/replay .sub files (Flipper compatible)
|
||||
- Spectrum analyzer
|
||||
- Frequency scanner
|
||||
- Signal recorder
|
||||
- SD card support for signature storage
|
||||
|
||||
**Installation**:
|
||||
```bash
|
||||
# Download from releases
|
||||
wget https://github.com/pr3y/Bruce/releases/latest/download/Bruce_T-Embed.bin
|
||||
|
||||
# Flash using esptool (in Termux)
|
||||
pip install esptool
|
||||
esptool.py --chip esp32s3 --port /dev/ttyUSB0 write_flash 0x0 Bruce_T-Embed.bin
|
||||
```
|
||||
|
||||
### 2. ESP32-Marauder
|
||||
**Repository**: https://github.com/justcallmekoko/ESP32Marauder
|
||||
|
||||
Originally for WiFi, has been adapted for Sub-GHz.
|
||||
|
||||
### 3. Custom Firmware (Planned)
|
||||
GigLez-specific firmware optimized for continuous scanning and GPS coordination.
|
||||
|
||||
## Communication Protocol
|
||||
|
||||
### Serial Connection
|
||||
|
||||
The T-Embed communicates via USB serial at 115200 baud.
|
||||
|
||||
**Termux Serial Setup**:
|
||||
```bash
|
||||
# Install required packages
|
||||
pkg install libusb python
|
||||
|
||||
# Python serial library
|
||||
pip install pyserial
|
||||
|
||||
# Find device
|
||||
ls /dev/ttyUSB*
|
||||
# Usually /dev/ttyUSB0 or /dev/ttyACM0
|
||||
```
|
||||
|
||||
### Command Protocol
|
||||
|
||||
GigLez uses a JSON-based command protocol over serial.
|
||||
|
||||
#### Command Structure
|
||||
|
||||
All commands are JSON objects terminated by `\n`:
|
||||
|
||||
```json
|
||||
{"cmd": "COMMAND_NAME", "params": {...}, "id": 12345}
|
||||
```
|
||||
|
||||
- `cmd`: Command name (uppercase)
|
||||
- `params`: Command-specific parameters
|
||||
- `id`: Optional request ID for response matching
|
||||
|
||||
#### Response Structure
|
||||
|
||||
```json
|
||||
{"status": "ok|error", "data": {...}, "id": 12345, "timestamp": 1704998400}
|
||||
```
|
||||
|
||||
- `status`: "ok" or "error"
|
||||
- `data`: Response data (command-specific)
|
||||
- `id`: Request ID (if provided)
|
||||
- `timestamp`: Unix timestamp from device
|
||||
|
||||
### Command Reference
|
||||
|
||||
#### 1. SCAN
|
||||
|
||||
Start scanning for signals on specified frequencies.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "SCAN",
|
||||
"params": {
|
||||
"frequencies": [433920000, 315000000, 868000000],
|
||||
"duration": 60,
|
||||
"modulation": "OOK",
|
||||
"rssi_threshold": -90
|
||||
},
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `frequencies`: Array of frequencies in Hz
|
||||
- `duration`: Scan duration in seconds (0 = continuous)
|
||||
- `modulation`: "OOK", "FSK", or "AUTO"
|
||||
- `rssi_threshold`: Minimum signal strength in dBm
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"scan_id": "scan_20250111_1234",
|
||||
"started_at": 1704998400
|
||||
},
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. CAPTURE
|
||||
|
||||
Capture a detected signal.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "CAPTURE",
|
||||
"params": {
|
||||
"frequency": 433920000,
|
||||
"duration": 5,
|
||||
"format": "RAW"
|
||||
},
|
||||
"id": 2
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `frequency`: Frequency in Hz
|
||||
- `duration`: Capture duration in seconds
|
||||
- `format`: "RAW", "DECODED", or "BOTH"
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"capture_id": "cap_20250111_1235",
|
||||
"frequency": 433920000,
|
||||
"rssi": -75,
|
||||
"protocol": "Princeton",
|
||||
"raw_data": [2980, -240, 520, -980, 520, -980, ...],
|
||||
"decoded": {
|
||||
"bit": 24,
|
||||
"key": "00 00 00 00 00 95 D5 D4",
|
||||
"te": 400
|
||||
}
|
||||
},
|
||||
"id": 2
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. STOP_SCAN
|
||||
|
||||
Stop an active scan.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "STOP_SCAN",
|
||||
"params": {
|
||||
"scan_id": "scan_20250111_1234"
|
||||
},
|
||||
"id": 3
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"total_signals": 127,
|
||||
"unique_protocols": 8
|
||||
},
|
||||
"id": 3
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. REPLAY
|
||||
|
||||
Replay a stored .sub file.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "REPLAY",
|
||||
"params": {
|
||||
"file": "/sd/captures/garage_door.sub",
|
||||
"repeat": 3,
|
||||
"delay_ms": 100
|
||||
},
|
||||
"id": 4
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"transmitted": 3,
|
||||
"frequency": 433920000
|
||||
},
|
||||
"id": 4
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. STATUS
|
||||
|
||||
Get device status.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "STATUS",
|
||||
"id": 5
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"firmware": "GigLez-T-Embed v0.1.0",
|
||||
"uptime": 3600,
|
||||
"battery_voltage": 3.85,
|
||||
"battery_percent": 78,
|
||||
"sd_card": {
|
||||
"present": true,
|
||||
"free_mb": 15234,
|
||||
"total_mb": 31456
|
||||
},
|
||||
"current_scan": "scan_20250111_1234",
|
||||
"cc1101": {
|
||||
"status": "idle",
|
||||
"frequency": 433920000,
|
||||
"rssi": -95
|
||||
}
|
||||
},
|
||||
"id": 5
|
||||
}
|
||||
```
|
||||
|
||||
#### 6. CONFIG
|
||||
|
||||
Configure device settings.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "CONFIG",
|
||||
"params": {
|
||||
"auto_save": true,
|
||||
"save_path": "/sd/giglez/",
|
||||
"led_brightness": 50,
|
||||
"spectrum_display": false
|
||||
},
|
||||
"id": 6
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"config_updated": true
|
||||
},
|
||||
"id": 6
|
||||
}
|
||||
```
|
||||
|
||||
#### 7. LIST_FILES
|
||||
|
||||
List captured .sub files on SD card.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "LIST_FILES",
|
||||
"params": {
|
||||
"path": "/sd/giglez/",
|
||||
"pattern": "*.sub"
|
||||
},
|
||||
"id": 7
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"files": [
|
||||
{
|
||||
"name": "capture_001.sub",
|
||||
"size": 245,
|
||||
"created": 1704998400
|
||||
},
|
||||
{
|
||||
"name": "capture_002.sub",
|
||||
"size": 312,
|
||||
"created": 1704998500
|
||||
}
|
||||
],
|
||||
"total_count": 2,
|
||||
"total_bytes": 557
|
||||
},
|
||||
"id": 7
|
||||
}
|
||||
```
|
||||
|
||||
#### 8. GET_FILE
|
||||
|
||||
Retrieve a file from SD card.
|
||||
|
||||
**Request**:
|
||||
```json
|
||||
{
|
||||
"cmd": "GET_FILE",
|
||||
"params": {
|
||||
"path": "/sd/giglez/capture_001.sub"
|
||||
},
|
||||
"id": 8
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"data": {
|
||||
"filename": "capture_001.sub",
|
||||
"size": 245,
|
||||
"content": "Filetype: Flipper SubGhz Key File\nVersion: 1\n..."
|
||||
},
|
||||
"id": 8
|
||||
}
|
||||
```
|
||||
|
||||
### Event Notifications
|
||||
|
||||
The T-Embed can send unsolicited event notifications:
|
||||
|
||||
#### SIGNAL_DETECTED
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "SIGNAL_DETECTED",
|
||||
"data": {
|
||||
"frequency": 433920000,
|
||||
"rssi": -72,
|
||||
"timestamp": 1704998400,
|
||||
"protocol": "Unknown",
|
||||
"duration_ms": 45
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### SCAN_COMPLETE
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "SCAN_COMPLETE",
|
||||
"data": {
|
||||
"scan_id": "scan_20250111_1234",
|
||||
"total_signals": 127,
|
||||
"duration": 60
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### ERROR
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "ERROR",
|
||||
"data": {
|
||||
"code": "SD_CARD_FULL",
|
||||
"message": "SD card is full, cannot save capture"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Python Communication Library
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
import serial
|
||||
import json
|
||||
import time
|
||||
|
||||
class TEmbedController:
|
||||
def __init__(self, port='/dev/ttyUSB0', baudrate=115200):
|
||||
self.serial = serial.Serial(port, baudrate, timeout=1)
|
||||
self.request_id = 0
|
||||
time.sleep(2) # Wait for device reset
|
||||
|
||||
def send_command(self, cmd, params=None):
|
||||
"""Send command and return response"""
|
||||
self.request_id += 1
|
||||
message = {
|
||||
'cmd': cmd,
|
||||
'params': params or {},
|
||||
'id': self.request_id
|
||||
}
|
||||
|
||||
# Send
|
||||
self.serial.write(json.dumps(message).encode() + b'\n')
|
||||
|
||||
# Wait for response
|
||||
line = self.serial.readline().decode().strip()
|
||||
if line:
|
||||
return json.loads(line)
|
||||
return None
|
||||
|
||||
def start_scan(self, frequencies, duration=60):
|
||||
"""Start scanning on specified frequencies"""
|
||||
return self.send_command('SCAN', {
|
||||
'frequencies': frequencies,
|
||||
'duration': duration,
|
||||
'modulation': 'AUTO',
|
||||
'rssi_threshold': -90
|
||||
})
|
||||
|
||||
def capture_signal(self, frequency, duration=5):
|
||||
"""Capture signal at frequency"""
|
||||
return self.send_command('CAPTURE', {
|
||||
'frequency': frequency,
|
||||
'duration': duration,
|
||||
'format': 'BOTH'
|
||||
})
|
||||
|
||||
def get_status(self):
|
||||
"""Get device status"""
|
||||
return self.send_command('STATUS')
|
||||
|
||||
def listen_events(self, callback):
|
||||
"""Listen for event notifications"""
|
||||
while True:
|
||||
line = self.serial.readline().decode().strip()
|
||||
if line:
|
||||
try:
|
||||
data = json.loads(line)
|
||||
if 'event' in data:
|
||||
callback(data)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Example usage
|
||||
device = TEmbedController('/dev/ttyUSB0')
|
||||
|
||||
# Check status
|
||||
status = device.get_status()
|
||||
print(f"Battery: {status['data']['battery_percent']}%")
|
||||
|
||||
# Start scan
|
||||
scan = device.start_scan([433920000, 315000000, 868000000], duration=60)
|
||||
print(f"Scan started: {scan['data']['scan_id']}")
|
||||
|
||||
# Listen for signals
|
||||
def on_event(event):
|
||||
if event['event'] == 'SIGNAL_DETECTED':
|
||||
print(f"Signal detected: {event['data']['frequency']} Hz, RSSI: {event['data']['rssi']} dBm")
|
||||
|
||||
device.listen_events(on_event)
|
||||
```
|
||||
|
||||
### Async Implementation
|
||||
|
||||
For better performance in production:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aioserial
|
||||
|
||||
class AsyncTEmbedController:
|
||||
def __init__(self, port='/dev/ttyUSB0', baudrate=115200):
|
||||
self.port = port
|
||||
self.baudrate = baudrate
|
||||
self.request_id = 0
|
||||
self.pending_requests = {}
|
||||
|
||||
async def connect(self):
|
||||
self.serial = aioserial.AioSerial(port=self.port, baudrate=self.baudrate)
|
||||
await asyncio.sleep(2)
|
||||
|
||||
async def send_command(self, cmd, params=None):
|
||||
self.request_id += 1
|
||||
message = {
|
||||
'cmd': cmd,
|
||||
'params': params or {},
|
||||
'id': self.request_id
|
||||
}
|
||||
|
||||
# Send
|
||||
await self.serial.write_async(json.dumps(message).encode() + b'\n')
|
||||
|
||||
# Create future for response
|
||||
future = asyncio.Future()
|
||||
self.pending_requests[self.request_id] = future
|
||||
|
||||
return await future
|
||||
|
||||
async def listen(self, event_callback):
|
||||
"""Background task to listen for responses and events"""
|
||||
while True:
|
||||
line = await self.serial.readline_async()
|
||||
if line:
|
||||
try:
|
||||
data = json.loads(line.decode().strip())
|
||||
|
||||
# Handle response
|
||||
if 'id' in data and data['id'] in self.pending_requests:
|
||||
self.pending_requests[data['id']].set_result(data)
|
||||
del self.pending_requests[data['id']]
|
||||
|
||||
# Handle event
|
||||
elif 'event' in data:
|
||||
await event_callback(data)
|
||||
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
```
|
||||
|
||||
## GPS Coordination
|
||||
|
||||
### Timestamp Synchronization
|
||||
|
||||
The T-Embed doesn't have GPS, so timestamps come from Android:
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
# Send timestamp to device
|
||||
device.send_command('CONFIG', {
|
||||
'system_time': int(time.time())
|
||||
})
|
||||
|
||||
# Device will use this to calibrate its internal clock
|
||||
```
|
||||
|
||||
### Capture with GPS
|
||||
|
||||
When a signal is captured, immediately tag it with GPS:
|
||||
|
||||
```python
|
||||
import android
|
||||
from tembed import TEmbedController
|
||||
|
||||
# Initialize Android SL4A for GPS
|
||||
droid = android.Android()
|
||||
droid.startLocating()
|
||||
|
||||
# T-Embed controller
|
||||
device = TEmbedController()
|
||||
|
||||
# Event handler
|
||||
async def on_signal(event):
|
||||
if event['event'] == 'SIGNAL_DETECTED':
|
||||
# Get current GPS location
|
||||
location = droid.getLastKnownLocation().result
|
||||
gps = location.get('gps') or location.get('network')
|
||||
|
||||
if gps:
|
||||
# Store in database with GPS
|
||||
store_capture({
|
||||
'frequency': event['data']['frequency'],
|
||||
'rssi': event['data']['rssi'],
|
||||
'latitude': gps['latitude'],
|
||||
'longitude': gps['longitude'],
|
||||
'accuracy': gps.get('accuracy', 0),
|
||||
'timestamp': event['data']['timestamp']
|
||||
})
|
||||
|
||||
# Start listening
|
||||
await device.listen(on_signal)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Device Not Detected
|
||||
|
||||
```bash
|
||||
# Check USB devices
|
||||
lsusb
|
||||
|
||||
# Check serial ports
|
||||
ls -l /dev/ttyUSB* /dev/ttyACM*
|
||||
|
||||
# Grant permissions (Termux)
|
||||
termux-usb -l # List devices
|
||||
termux-usb -r /dev/bus/usb/XXX/YYY # Request permission
|
||||
```
|
||||
|
||||
### Serial Communication Errors
|
||||
|
||||
```python
|
||||
# Add error handling
|
||||
try:
|
||||
response = device.send_command('STATUS')
|
||||
except serial.SerialException as e:
|
||||
print(f"Serial error: {e}")
|
||||
# Attempt reconnection
|
||||
device.serial.close()
|
||||
device.serial.open()
|
||||
```
|
||||
|
||||
### Firmware Update
|
||||
|
||||
```bash
|
||||
# Backup SD card first
|
||||
adb pull /sdcard/giglez /path/to/backup
|
||||
|
||||
# Flash new firmware
|
||||
esptool.py --chip esp32s3 --port /dev/ttyUSB0 erase_flash
|
||||
esptool.py --chip esp32s3 --port /dev/ttyUSB0 write_flash 0x0 firmware.bin
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Flash Bruce firmware to T-Embed
|
||||
2. Test serial communication from Termux
|
||||
3. Implement capture coordination with GPS
|
||||
4. Build async event handling system
|
||||
5. Develop custom firmware optimized for GigLez
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user