Initial public release

Full BigBrother network implant - passive SOC + active exploitation.
Personal identifiers removed; all capabilities intact.
See README.md for setup and docs/deployment.md for detailed deployment.
This commit is contained in:
n0mad1k
2026-06-26 09:52:50 -04:00
commit ccc6b729de
175 changed files with 47941 additions and 0 deletions
+277
View File
@@ -0,0 +1,277 @@
---
agent: fix-planner
status: COMPLETE
timestamp: 2026-04-10T13:45:00Z
total_findings_raw: 32
total_findings_deduped: 29
p1_count: 4
p2_count: 6
p3_count: 12
p4_count: 7
devtrack_items_created: [551, 552, 553, 554, 555, 556, 557, 558, 559, 560]
errors: []
---
# Presence Daemon Fix Plan
## Deduplication Notes
- **Raw findings:** 32 across 6 audit files (code-auditor, security-auditor, env-validator, db-auditor, test-runner, impl notes)
- **Deduped:** 29 unique issues (merged findings with same file + issue type; highest severity retained; sources cited)
- **Sources combined:** DB connection leak (2 sources), struct unpacking edge cases (2 sources), BLE error recovery (2 sources)
---
## P1 — Block Deploy (CRITICAL/HIGH Security or Correctness)
- [x] **[CRITICAL]** `presence_daemon.py:648-656` HTTP status server port hardcoded (9191) — acts as fingerprint | Sources: security-auditor | Effort: XS | DevTrack: #551
- **Context:** Port discoverable via `ss -tlnp`, identifies tool via process enumeration
- **Fix:** Make configurable via `PRESENCE_STATUS_PORT` env var; document per-deployment variation
- **Acceptance:** Port reads from env var; default fallback to 9191 if unset
- **Status:** FIXED — STATUS_PORT = int(os.getenv("PRESENCE_STATUS_PORT", "9191")) added at L31; status_server() updated to use STATUS_PORT; install.sh comments updated
- [x] **[CRITICAL]** `presence_daemon.py:156, 172, 193, 208` SQLite connection leak in exception handlers | Sources: db-auditor | Effort: S | DevTrack: #552
- **Context:** Functions `lookup_person()`, `seed_persons_from_db()`, `log_signal()`, `log_presence_change()` don't close connections on exceptions; resource exhaustion under high-frequency signal logging
- **Fix:** Wrap all `sqlite3.connect()` in try/finally or use context manager (`with` statement)
- **Acceptance:** All four functions use context manager; no unclosed connection paths
- **Status:** FIXED — All four functions wrapped with try/finally and conn.close() in finally block; timeout increased to 10s
- [x] **[HIGH]** `presence_daemon.py:272-323` Race condition in `update_person_state()` — two-lock acquisition | Sources: code-auditor | Effort: S | DevTrack: #553
- **Context:** Lock released between `devices_lock` and `persons_lock`; `max_last_seen` snapshot becomes stale. Concurrent signal could change person state after certainty computed but before lock acquired, causing duplicate alerts or missed transitions.
- **Fix:** Acquire both locks atomically or compute entire state transition inside single critical section; snapshot under lock before computing state machine
- **Acceptance:** State machine transitions (UNKNOWN→PRESENT, PRESENT→ABSENT) protected by single lock; no log entries showing duplicate/missed transitions
- **Status:** FIXED — Certainty computed under devices_lock and snapshot held; state machine logic moved inside persons_lock; alert sending moved outside locks to minimize critical section
- [ ] **[HIGH]** `presence_daemon.py:384-404` Raw socket struct unpacking lacks full validation | Sources: security-auditor | Effort: S | DevTrack: #554
- **Context:** Netlink parsing checks `nlmsg_len < 16` but not actual buffer length; DHCP MAC extraction at `[28:34]` not validated. Crafted packets could cause silent truncation, incorrect MAC parsing, or exception flood.
- **Fix:** Add explicit length checks before struct.unpack_from() and MAC extraction:
```python
# Netlink: before struct.unpack_from('=IHHII', data, offset)
if len(data) < offset + 16:
logger.warning("Truncated netlink message")
return
# DHCP: before mac_bytes = dhcp[28:34]
if len(dhcp) < 34:
return
```
- **Acceptance:** All edge cases validated; unit tests (if added) should cover truncated packets
- **Status:** NOT FIXED (out of scope for this bugfixer task — only 4 P1 items assigned)
---
## P2 — Fix This Week (HIGH Code Quality or Significant Risk)
- [ ] **[HIGH]** `presence_daemon.py:289` Magic number in aggregate certainty calculation | Sources: code-auditor | Effort: XS | DevTrack: #555
- **Context:** Hardcoded `0.08` coefficient for secondary device weighting (line 289: `certainties[0] + 0.08 * certainties[1]`) should be named constant for clarity and maintainability
- **Fix:** Extract to module-level constant:
```python
SECONDARY_DEVICE_WEIGHT = 0.08
# Then in update_person_state():
aggregate_certainty = certainties[0] + SECONDARY_DEVICE_WEIGHT * certainties[1]
```
- **Acceptance:** Constant named, value unchanged, formula still produces same results
- [ ] **[HIGH]** `presence_daemon.py:20` Unused import urllib.parse | Sources: code-auditor | Effort: XS | DevTrack: #556
- **Context:** Imported but never used; only `urllib.request.Request()` and `urllib.request.urlopen()` are called
- **Fix:** Remove line 20: `import urllib.parse`
- **Acceptance:** No import errors; linter reports no unused imports
- [ ] **[HIGH]** `install.sh:31` Database path mismatch (install.sh vs daemon defaults) | Sources: code-auditor | Effort: XS | DevTrack: #557
- **Context:** install.sh forces `/opt/sensor/sensor.db` (line 31) but daemon defaults to `/opt/presence/presence.db` (presence_daemon.py:29). Will create two separate databases.
- **Fix:** Align paths. Recommend standardizing on `/opt/presence/presence.db` (more descriptive name):
- Update install.sh line 31: `PRESENCE_DB_PATH=/opt/presence/presence.db`
- Verify daemon defaults match or accept via env var override
- **Acceptance:** Single database file created; systemd EnvironmentFile sets consistent path; no duplicate databases on disk
- [ ] **[HIGH]** `presence_daemon.py:541-573` BLE scanner missing error recovery — event loop can die silently | Sources: security-auditor, db-auditor | Effort: M | DevTrack: #558
- **Context:** No retry loop or watchdog; malformed BLE packets could crash event loop, silently disabling BLE tracking. Also overlaps with seed_persons_from_db() error handling (db-auditor: missing recovery on empty persons table).
- **Fix:**
1. Add retry loop with exponential backoff to `ble_scanner()` thread:
```python
retries = 0
while retries < 5:
try:
loop.run_until_complete(run_ble_scan())
retries = 0 # Reset on success
except Exception as e:
logger.error(f"BLE scan iteration failed: {e}")
retries += 1
time.sleep(2 ** retries) # Exponential backoff: 2, 4, 8, 16, 32 sec
logger.critical("BLE scanner exhausted retries; disabling")
```
2. Implement watchdog thread to detect BLE thread death and restart
3. Validate persons table on startup in `seed_persons_from_db()`: fail loud if empty and no override
- **Acceptance:** BLE scanner recovers from transient errors; critical log on exhaustion; daemon validates person list at startup
- [x] **[HIGH]** `presence_daemon.py:142-147` init_db() missing WAL mode and pragmas | Sources: db-auditor | Effort: XS | DevTrack: #559
- **Context:** No PRAGMA journal_mode=WAL; synchronous journaling under 6-thread concurrent write load will serialize and timeout (SQLITE_BUSY errors)
- **Fix:** Add to `init_db()` or presence_schema.sql:
```python
# In init_db(), after conn.execute(schema):
conn.execute('PRAGMA journal_mode=WAL')
conn.execute('PRAGMA synchronous=NORMAL')
conn.execute('PRAGMA foreign_keys=ON')
conn.commit()
```
- **Acceptance:** `PRAGMA journal_mode` reports WAL; no SQLITE_BUSY timeouts under sustained signal load; schema integrity enforced
- **Status:** FIXED — WAL pragmas added to init_db() at L148-150, committed at L151
- [ ] **[HIGH]** `presence_daemon.py:605` N+1 pattern in decay_loop — inefficient person state recomputation | Sources: db-auditor | Effort: M | DevTrack: #560
- **Context:** Every person state update re-computes aggregate certainty from full devices dict; should batch-compute all person states once per decay interval
- **Fix:** Refactor decay_loop to compute all person states in one pass:
```python
def decay_loop():
while True:
time.sleep(60)
try:
with devices_lock:
# Collect all person_ids
person_ids = set(d.person_id for d in devices.values() if d.person_id)
# Compute all person states in one critical section
decayed_devices = {mac: decay_device(dev) for mac, dev in devices.items()}
# State transitions outside lock
for person_id in person_ids:
update_person_state(person_id)
```
- **Acceptance:** Decay loop computes person states once per 60-second interval; no duplicate iterations per person
---
## P3 — Fix This Month (MEDIUM Code Quality or Noticeable Risk)
- [ ] **[MEDIUM]** `presence_daemon.py:468-538` High cyclomatic complexity in parse_dhcp() | Sources: code-auditor | Effort: S
- Description: 71 lines, 20 branches, 5 nesting levels; mixes header parsing + DHCP option parsing
- Fix: Extract packet validation to separate function; move DHCP option parsing to dedicated helper
- Remediation: `parse_dhcp_options(dhcp_payload) -> dict` returning option_type → value mapping
- [ ] **[MEDIUM]** `presence_daemon.py:326-361` Deep nesting in probe_sniffer() — 6 levels | Sources: code-auditor | Effort: S
- Description: Multiple if statements nested 6 levels; hard to follow control flow
- Fix: Extract frame validation logic into `validate_probe_request(data) -> Optional[str]` returning MAC or None
- [ ] **[MEDIUM]** `presence_daemon.py:576-609` Inconsistent error handling in decay_loop() | Sources: code-auditor | Effort: S
- Description: Catches all exceptions but silently continues; no distinction between transient and fatal errors
- Fix: Separate transient network errors (debug level) from state machine errors (error/warning level)
- [ ] **[MEDIUM]** `presence_daemon.py:282-284` Early return in update_person_state() leaves persons_lock unprotected | Sources: code-auditor | Effort: XS
- Description: If no devices found, returns inside devices_lock but person state not re-checked
- Fix: Re-check person state after lock release or defer return until both locks acquired
- [ ] **[MEDIUM]** `presence_daemon.py:437-439` Weak infrastructure IP filtering — empty set never populated | Sources: code-auditor | Effort: S
- Description: infrastructure_ips checked but never initialized; filter never applies
- Fix: Either populate from config at startup or remove dead code
- [ ] **[MEDIUM]** `presence_daemon.py:661-686` Daemon thread join() semantics — daemon=True threads not blocking | Sources: code-auditor | Effort: S
- Description: All threads created with daemon=True then joined; join() is vestigial
- Fix: Remove daemon=True flag (prefer explicit cleanup) or remove join() calls
- [ ] **[MEDIUM]** `presence_daemon.py:43-50` Logging fingerprints — tool identity in log messages | Sources: security-auditor | Effort: S
- Description: Messages like "Sensor daemon started", "Probe sniffer", "DHCP sniffer" identify daemon as surveillance tool
- Fix: Use generic messages ("Network listener probe active", "Listening for ARP events"); move identification to comments only
- [ ] **[MEDIUM]** `presence_daemon.py:296-323` HTTP webhook timeout under lock — blocks person state queries | Sources: security-auditor | Effort: S
- Description: persons_lock held during send_alert(); 5-second HTTP timeout blocks all state reads
- Fix: Snapshot person data under lock, release before send_alert():
```python
with persons_lock:
person = persons.get(person_id)
if person is None:
return
old_status = person.status
# ... state machine ...
person.status = new_status
# Outside lock
send_alert(person.name, "PRESENT", ...)
```
- [ ] **[MEDIUM]** `install.sh:20-35` Install script missing Matrix webhook configuration | Sources: env-validator | Effort: S
- Description: Systemd unit does not include PRESENCE_MATRIX_WEBHOOK; users must manually add
- Fix: Add Infisical integration; create start.sh wrapper that fetches webhook from vault at boot
- Remediation: `eval $(creds env bigbrother)` pattern in start.sh
- [ ] **[MEDIUM]** `presence_schema.sql:22` Missing index on occupancy_log(ts) | Sources: db-auditor | Effort: XS
- Description: High-frequency writes without ts index; time-range queries will full-table scan
- Fix: Add `CREATE INDEX idx_occupancy_ts ON occupancy_log(ts);`
- [ ] **[MEDIUM]** `presence_daemon.py:156, 172, 193, 208` Connection timeout too short (1 second) | Sources: db-auditor | Effort: XS
- Description: Under multi-threaded contention + synchronous journaling, SQLITE_BUSY errors likely
- Fix: Increase timeout to 10-30 seconds in all sqlite3.connect(timeout=...) calls OR enable WAL + NORMAL synchronous (via P2 #559)
- [ ] **[MEDIUM]** `presence_daemon.py` No tests exist for presence daemon | Sources: test-runner | Effort: L
- Description: Zero test coverage for 690-line daemon with 15+ functions
- Note: Per task instructions, tests not created by test-runner. Flag for human review if required as P1 acceptance criterion.
---
## P4 — Backlog (LOW Code Quality or Style)
- [ ] **[LOW]** `presence_daemon.py:601` Inefficient device pruning during iteration | Effort: XS
- Description: Deletes from dict during iteration (safe via deferred list but inelegant)
- Fix: Build new dict with comprehension: `devices = {mac: dev for mac, dev in devices.items() if elapsed_minutes < 120}`
- [ ] **[LOW]** `presence_daemon.py:612-614` Inline handler factory pattern creates class per request | Effort: XS
- Description: StatusHandler class defined in function, returns new class per request
- Fix: Define StatusHandler at module level, pass to HTTPServer
- [ ] **[LOW]** `presence_daemon.py:241-270` Bare on_signal() calls lack source context | Effort: S
- Description: Called from 5 sources but doesn't log caller/thread context
- Fix: Add logger context or thread names for debugging multi-threaded flow
- [ ] **[LOW]** `presence_schema.sql:1-34` No compound index on frequent queries | Effort: XS
- Description: idx_signals_mac and idx_signals_ts separate but queries filter on both
- Fix: Add `CREATE INDEX idx_signals_mac_ts ON signals(mac, ts);`
- [ ] **[LOW]** `install.sh:10-11` Unnecessary chown after mkdir | Effort: XS
- Description: mkdir with sudo, then chown may fail or be no-op
- Fix: Run entire install non-root with sudo only for systemd registration
- [ ] **[LOW]** `presence_schema.sql:1-28` Schema lacks enforced foreign keys | Effort: XS
- Description: REFERENCES present but PRAGMA foreign_keys never enabled
- Fix: Execute `PRAGMA foreign_keys=ON;` in init_db() before schema execution
- [ ] **[LOW]** `presence_daemon.py:142-147` No connection validation in init_db() | Effort: S
- Description: Never verifies schema creation succeeded or tables readable
- Fix: Add startup check to validate tables exist and are readable
---
## Summary Table
| Priority | Count | Effort | DevTrack | Status |
|----------|-------|--------|----------|--------|
| P1 | 4 | 3×S, 1×XS | #551-554 | BLOCKED until fixed |
| P2 | 6 | 2×M, 3×S, 1×XS | #555-560 | BLOCKED until fixed |
| P3 | 12 | 1×L, 8×S, 3×XS | — | Can run tests after P1+P2 |
| P4 | 7 | 1×S, 6×XS | — | Nice to have |
---
## Effort Breakdown
- **XS** (<30 min): 8 items (magic number, unused import, path mismatch, connection timeout, occupancy index, compound index, chown, foreign keys)
- **S** (30 min - 2 hr): 12 items (connection leak, race condition, struct validation, error handling, logging, webhook, deep nesting, nesting, etc.)
- **M** (2 - 8 hr): 2 items (BLE recovery, N+1 pattern, WAL mode + pragmas)
- **L** (1 - 3 days): 1 item (test suite)
**Minimal path to unblock deploy (P1+P2):** ~10 hours effort across 10 critical fixes.
---
## Deployment Gate
**Item #530 cannot advance to testing until:**
1. All P1 items (#551-554) fixed and verified
2. All P2 items (#555-560) fixed and verified
3. Code review confirms no regressions in:
- Signal processing pipeline
- Person state machine transitions
- Database persistence and recovery
- Thread safety (no new deadlocks)
**Testing verification checklist:**
- [ ] No SQLITE_BUSY errors under sustained signal load
- [ ] No connection exhaustion after 1 hour runtime
- [ ] Person state transitions fire correctly (no duplicates, no missed alerts)
- [ ] Status endpoint accessible on configurable port
- [ ] Database path matches install.sh defaults or env override
- [ ] BLE scanner recovers from transient errors
+202
View File
@@ -0,0 +1,202 @@
# Presence Daemon — Person Presence Intelligence
Passive multi-signal WiFi/ARP/DHCP/BLE person presence tracking daemon for BigBrother drop implant.
## Quick Start
```bash
cd /path/to/bigbrother/presence
sudo bash install.sh
```
This creates `/opt/sensor/sensor.py`, initializes the database, and starts the systemd service.
## Configuration
Environment variables:
- `PRESENCE_MONITOR_IFACE` — WiFi monitor mode interface (default: wlan1)
- `PRESENCE_DATA_IFACE` — Connected network interface for ARP/DHCP (default: wlan0)
- `PRESENCE_DB_PATH` — SQLite database location (default: /opt/presence/presence.db)
- `PRESENCE_MATRIX_WEBHOOK` — Matrix webhook URL for alerts (default: empty, no alerts)
Example systemd override:
```bash
sudo systemctl edit sensor
# Add to [Service] section:
# Environment=PRESENCE_MONITOR_IFACE=wlan2
# Environment=PRESENCE_MATRIX_WEBHOOK=https://matrix.example.com/hook/...
```
## Architecture
### Sensor Threads (4 concurrent)
1. **Probe Sniffer** — WiFi probe request capture on monitor interface
- Passive 802.11 frame parsing (type 0, subtype 4)
- Extracts source MAC → +0.60 certainty bump
- Runs on PRESENCE_MONITOR_IFACE
2. **ARP Listener** — netlink RTM_NEWNEIGH/RTM_DELNEIGH events
- Kernel neighbor discovery events
- Filters infrastructure IPs
- Extracts MAC → +0.50 certainty bump
3. **DHCP Sniffer** — AF_PACKET raw socket on data interface
- Captures DHCP DISCOVER/REQUEST packets
- Extracts client MAC (chaddr field)
- Extracts hostname (option 12)
- MAC → +0.80 certainty bump
4. **BLE Scanner** — Passive BLE device discovery
- Uses `bleak` library if available
- Gracefully disables if ImportError
- Device address → +0.40 certainty bump
### Core Logic
**Signal Fusion** (per person):
- Collect all device certainties
- Apply exponential decay: `certainty × e^(-0.08 × age_minutes)`
- Aggregate: `max + 0.08 × second_max`
- Check state machine
**Presence Anchor** (critical):
- Once PRESENT (certainty ≥ 0.40), hold PRESENT for 45 minutes from last signal
- Prevents false ABSENT from temporary signal loss (iOS suppression, WiFi roam, power save)
- After 45 minutes without signal ≥ 0.40 → ABSENT
**State Machine**:
```
UNKNOWN → PRESENT → ABSENT → PRESENT
↓ agg_cert ↓
└─────────≥ 0.40 45 min no signal
```
**Alerting**:
- UNKNOWN → PRESENT: "ARRIVED" alert
- PRESENT → ABSENT: "DEPARTED" alert
- Via Matrix webhook if configured
### Database Schema
**persons** — Named people
- id (PK)
- name — Person name
- notes — Optional metadata
**devices** — MAC-to-person mapping
- mac (PK)
- person_id (FK)
- label — Device name (iPhone, Laptop, etc.)
- added_at — Timestamp
**signals** — Raw signal observations
- id (PK)
- mac — Device MAC
- signal_type — "probe", "arp", "dhcp", or "ble"
- certainty — Aggregated certainty after this signal
- ts — Observation timestamp
**occupancy_log** — State transitions
- id (PK)
- person_id (FK)
- old_status — Previous state
- new_status — New state
- ts — Transition timestamp
## HTTP Status Endpoint
GET http://127.0.0.1:9191/
Returns JSON:
```json
{
"persons": [
{
"id": 1,
"name": "Alice",
"status": "PRESENT",
"last_change": 1712761234.5
}
]
}
```
## Limitations (Phase 1)
- **No auto-enrollment**: Device-to-person mapping must be set up manually in database
- **No mDNS parsing**: Hostnames are not extracted or used for identity
- **BLE optional**: Requires `bleak` library; gracefully disabled if missing
- **Monitor mode required**: Probe sniffer needs separate WiFi adapter in monitor mode
- **Root access needed**: Raw sockets require CAP_NET_ADMIN or root
- **Single location**: No multi-room support
## Files
- `presence_daemon.py` — Main daemon (1000+ lines)
- `presence_schema.sql` — SQLite schema
- `install.sh` — Deployment script
- `AUDIT_impl.md` — Implementation notes and audit checklist
- `README.md` — This file
## Logging
Logs to `/var/log/sensor.log` and stdout. Log level: INFO (debug messages only on errors).
Example log output:
```
2026-04-10 14:54:32,123 [INFO] Database initialized at /opt/sensor/sensor.db
2026-04-10 14:54:32,124 [INFO] Loaded 2 persons from database
2026-04-10 14:54:32,125 [INFO] Probe sniffer started on wlan1
2026-04-10 14:54:32,126 [INFO] ARP listener started
2026-04-10 14:54:32,127 [INFO] DHCP sniffer started on wlan0
2026-04-10 14:54:32,128 [INFO] BLE scanner started
2026-04-10 14:54:32,129 [INFO] Status server started on http://127.0.0.1:9191
2026-04-10 14:54:32,130 [INFO] Sensor daemon started
```
## Testing
1. Verify systemd service is running:
```bash
sudo systemctl status sensor
```
2. Check database initialization:
```bash
sqlite3 /opt/sensor/sensor.db ".tables"
```
3. Query persons:
```bash
sqlite3 /opt/sensor/sensor.db "SELECT * FROM persons;"
```
4. Query recent signals:
```bash
sqlite3 /opt/sensor/sensor.db "SELECT * FROM signals ORDER BY ts DESC LIMIT 10;"
```
5. Check status endpoint:
```bash
curl http://127.0.0.1:9191/
```
6. Check logs:
```bash
tail -f /var/log/sensor.log
```
## Phase 2 Roadmap
- Auto-enrollment from mDNS hostnames
- BLE MAC signature recognition (Apple Watch, AirPods)
- Device quorum (multi-device presence confirmation)
- Tentative device linking via co-occurrence windows
## Phase 3 Roadmap
- Pattern of life analytics
- Arrival/departure time learning
- Occupancy forecasting
- Multi-location aggregation
+50
View File
@@ -0,0 +1,50 @@
#!/bin/bash
# Install script for presence daemon
# Deploys sensor.py and service configuration to /opt/sensor/
set -e
echo "Installing presence daemon..."
# Create sensor directory
sudo mkdir -p /opt/sensor
sudo chown $USER:$USER /opt/sensor
# Copy daemon and schema
cp presence_daemon.py /opt/sensor/sensor.py
cp presence_schema.sql /opt/sensor/
chmod +x /opt/sensor/sensor.py
echo "Created /opt/sensor/sensor.py and schema"
# Create systemd service
sudo tee /etc/systemd/system/sensor.service > /dev/null << 'EOF'
[Unit]
Description=Network Sensor
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/sensor/sensor.py
Restart=on-failure
RestartSec=10
Environment=PRESENCE_DB_PATH=/opt/sensor/presence.db
# Optional: override default status port (9191)
# Environment=PRESENCE_STATUS_PORT=9191
# Optional: configure Matrix webhook for alerts
# Environment=PRESENCE_MATRIX_WEBHOOK=https://matrix.example.com/hook
[Install]
WantedBy=multi-user.target
EOF
echo "Created /etc/systemd/system/sensor.service"
# Reload systemd and enable service
sudo systemctl daemon-reload
sudo systemctl enable sensor
sudo systemctl start sensor
echo "Enabled and started sensor service"
echo "Status:"
sudo systemctl status sensor --no-pager | head -20
+719
View File
@@ -0,0 +1,719 @@
#!/usr/bin/env python3
"""
Presence daemon — Multi-signal WiFi/ARP/DHCP/BLE person presence tracking.
Passive sensors with exponential decay and presence anchoring.
Zero active probing. No tool fingerprints in output.
"""
import asyncio
import http.server
import json
import logging
import math
import os
import socket
import sqlite3
import struct
import sys
import threading
import time
import urllib.parse
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
# --- Config (env-driven) ---
MONITOR_IFACE = os.getenv("PRESENCE_MONITOR_IFACE", "wlan1")
DATA_IFACE = os.getenv("PRESENCE_DATA_IFACE", "wlan0")
DB_PATH = os.getenv("PRESENCE_DB_PATH", "/opt/sensor/presence.db")
MATRIX_WEBHOOK = os.getenv("PRESENCE_MATRIX_WEBHOOK", "")
STATUS_PORT = int(os.getenv("PRESENCE_STATUS_PORT", "9191"))
PRESENCE_THRESHOLD = 0.40
ABSENCE_THRESHOLD = 0.10
ANCHOR_MINUTES = 45
DECAY_LAMBDA = 0.08
# Signal bump constants
PROBE_BUMP = 0.60
ARP_BUMP = 0.50
DHCP_BUMP = 0.80
BLE_BUMP = 0.40
# Logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("/var/log/sensor.log", encoding="utf-8"),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
# Top OUI prefixes (inline, no external DB)
OUI_DICT = {
"88A29E": "Apple",
"001A7D": "Apple",
"185F3F": "Apple",
"A4C3F0": "Apple",
"0899D8": "Apple",
"004096": "Apple",
"00219B": "Apple",
"0021E9": "Apple",
"005973": "Apple",
"006377": "Apple",
"0064B9": "Apple",
"0084F3": "Apple",
"00A04D": "Apple",
"00D04B": "Apple",
"28879F": "Google",
"2887BA": "Google",
"5427EB": "Google",
"542758": "Google",
"341513": "Amazon",
"0C47C2": "Amazon",
"B827EB": "Raspberry Pi",
"2C56DC": "Raspberry Pi",
"E45F01": "Raspberry Pi",
}
# Netlink constants
NETLINK_ROUTE = 0
RTMGRP_NEIGH = 0x4
RTM_NEWNEIGH = 28
RTM_DELNEIGH = 29
NUD_REACHABLE = 0x02
NUD_STALE = 0x04
NUD_DELAY = 0x08
NUD_PROBE = 0x10
NDA_DST = 1
NDA_LLADDR = 2
@dataclass
class DeviceState:
"""In-memory device state."""
mac: str
certainty: float = 0.0
last_seen: float = field(default_factory=time.time)
last_signal: str = ""
person_id: Optional[int] = None
@dataclass
class PersonState:
"""In-memory person state."""
person_id: int
name: str
status: str = "UNKNOWN"
last_change: float = field(default_factory=time.time)
# Shared state with locks
devices = {}
devices_lock = threading.Lock()
persons = {}
persons_lock = threading.Lock()
infrastructure_ips = set()
infrastructure_lock = threading.Lock()
def lookup_oui(mac: str) -> str:
"""Look up OUI vendor from MAC (first 3 octets, inline dict)."""
if not mac or mac == "00:00:00:00:00:00":
return "unknown"
oui_prefix = mac.replace(":", "").upper()[:6]
return OUI_DICT.get(oui_prefix, "unknown")
def init_db() -> None:
"""Initialize SQLite database from schema if not exists."""
schema_path = Path(__file__).parent / "presence_schema.sql"
if not schema_path.exists():
logger.error(f"Schema file not found: {schema_path}")
return
db_path = Path(DB_PATH)
db_path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(DB_PATH, timeout=10)
try:
cursor = conn.cursor()
schema = schema_path.read_text()
cursor.executescript(schema)
conn.commit()
cursor.execute("PRAGMA journal_mode=WAL")
cursor.execute("PRAGMA synchronous=NORMAL")
cursor.execute("PRAGMA foreign_keys=ON")
conn.commit()
logger.info(f"Database initialized at {DB_PATH}")
except Exception as e:
logger.error(f"Failed to init database: {e}")
finally:
conn.close()
def lookup_person(mac: str) -> Optional[int]:
"""Look up person_id for a MAC from database."""
conn = sqlite3.connect(DB_PATH, timeout=10)
try:
cursor = conn.cursor()
result = cursor.execute(
"SELECT person_id FROM devices WHERE mac = ?",
(mac.lower(),)
).fetchone()
return result[0] if result else None
except Exception as e:
logger.debug(f"lookup_person error for {mac}: {e}")
return None
finally:
conn.close()
def seed_persons_from_db() -> None:
"""Load all persons from database into memory."""
conn = sqlite3.connect(DB_PATH, timeout=10)
try:
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
rows = cursor.execute("SELECT id, name FROM persons").fetchall()
with persons_lock:
for row in rows:
person = PersonState(
person_id=row["id"],
name=row["name"]
)
persons[row["id"]] = person
logger.info(f"Loaded {len(rows)} persons from database")
except Exception as e:
logger.error(f"Failed to seed persons: {e}")
finally:
conn.close()
def log_signal(mac: str, signal_type: str, certainty: float) -> None:
"""Insert signal into database (non-blocking via queue would be ideal, but inline for simplicity)."""
conn = sqlite3.connect(DB_PATH, timeout=10)
try:
cursor = conn.cursor()
cursor.execute(
"INSERT INTO signals (mac, signal_type, certainty, ts) VALUES (?, ?, ?, ?)",
(mac.lower(), signal_type, certainty, time.time())
)
conn.commit()
except Exception as e:
logger.debug(f"Failed to log signal: {e}")
finally:
conn.close()
def log_presence_change(person_id: int, old_status: str, new_status: str) -> None:
"""Insert occupancy change into database."""
conn = sqlite3.connect(DB_PATH, timeout=10)
try:
cursor = conn.cursor()
cursor.execute(
"INSERT INTO occupancy_log (person_id, old_status, new_status, ts) VALUES (?, ?, ?, ?)",
(person_id, old_status, new_status, time.time())
)
conn.commit()
except Exception as e:
logger.debug(f"Failed to log presence change: {e}")
finally:
conn.close()
def send_alert(person_name: str, status: str, signal_type: str, certainty: float) -> None:
"""Send alert to Matrix webhook if configured."""
if not MATRIX_WEBHOOK:
logger.info(f"Alert (no webhook): {person_name}{status} ({signal_type}, certainty: {certainty:.2f})")
return
try:
action = "ARRIVED" if status == "PRESENT" else "DEPARTED"
message = f"SENSOR: {person_name}{action} ({signal_type}, certainty: {certainty:.2f})"
payload = json.dumps({"text": message})
req = urllib.request.Request(
MATRIX_WEBHOOK,
data=payload.encode("utf-8"),
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=5) as response:
logger.info(f"Alert sent: {message}")
except Exception as e:
logger.error(f"Failed to send alert: {e}")
def on_signal(mac: str, signal_type: str, bump: float) -> None:
"""
Process a signal bump for a device.
Decay old certainty, apply bump, update state.
"""
mac = mac.lower()
with devices_lock:
dev = devices.get(mac)
if dev is None:
dev = DeviceState(
mac=mac,
certainty=0.0,
last_seen=time.time(),
last_signal="",
person_id=lookup_person(mac)
)
devices[mac] = dev
# Apply exponential decay since last signal
elapsed_minutes = (time.time() - dev.last_seen) / 60.0
dev.certainty = dev.certainty * math.exp(-DECAY_LAMBDA * elapsed_minutes)
# Apply bump
dev.certainty = min(1.0, dev.certainty + bump)
dev.last_seen = time.time()
dev.last_signal = signal_type
log_signal(mac, signal_type, dev.certainty)
update_person_state(dev.person_id)
def update_person_state(person_id: Optional[int]) -> None:
"""
Compute aggregate certainty for a person, update state machine.
Fire alerts on state transitions.
"""
if person_id is None:
return
# Snapshot device state under lock
with devices_lock:
# Get all devices for this person
devices_for_person = [d for d in devices.values() if d.person_id == person_id]
if not devices_for_person:
return
# Compute aggregate certainty (max + 0.08 * second_max)
certainties = sorted([d.certainty for d in devices_for_person], reverse=True)
if len(certainties) >= 2:
agg_certainty = certainties[0] + 0.08 * certainties[1]
else:
agg_certainty = certainties[0]
# Get max last_seen time
max_last_seen = max(d.last_seen for d in devices_for_person)
# Determine new status based on snapshot
current_time = time.time()
elapsed_since_signal = (current_time - max_last_seen) / 60.0
with persons_lock:
person = persons.get(person_id)
if person is None:
return
old_status = person.status
new_status = old_status
# State machine logic
if person.status == "UNKNOWN":
if agg_certainty >= PRESENCE_THRESHOLD:
new_status = "PRESENT"
elif person.status == "PRESENT":
# Check presence anchor
if elapsed_since_signal >= ANCHOR_MINUTES:
new_status = "ABSENT"
elif person.status == "ABSENT":
if agg_certainty >= PRESENCE_THRESHOLD:
new_status = "PRESENT"
# Apply state transition if changed
if new_status != old_status:
person.status = new_status
person.last_change = current_time
log_presence_change(person_id, old_status, new_status)
# Send alert outside of lock
if new_status != old_status:
if new_status == "PRESENT":
send_alert(person.name, "PRESENT", "signal", agg_certainty)
else:
send_alert(person.name, "ABSENT", "timeout", agg_certainty)
def probe_sniffer(iface: str) -> None:
"""
WiFi probe request sniffer.
Raw 802.11 frame capture on monitor mode interface.
Extracts source MAC from probe requests.
"""
try:
s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(0x0003))
s.bind((iface, 0))
logger.info(f"Probe sniffer started on {iface}")
while True:
try:
data, _ = s.recvfrom(65535)
if len(data) < 24:
continue
# 802.11 frame: FC (2) + duration (2) + dest (6) + src (6) + BSSID (6) = 22 bytes min
fc = struct.unpack("<H", data[0:2])[0]
frame_type = (fc >> 2) & 0x3
frame_subtype = (fc >> 4) & 0xf
# Type 0 = management, Subtype 4 = probe request
if frame_type == 0 and frame_subtype == 4:
if len(data) >= 16:
# Source MAC at bytes 10-16
mac_bytes = data[10:16]
mac = ':'.join(f'{b:02x}' for b in mac_bytes)
if mac != "ff:ff:ff:ff:ff:ff" and mac != "00:00:00:00:00:00":
on_signal(mac, "probe", PROBE_BUMP)
except Exception as e:
logger.debug(f"Probe sniffer error: {e}")
time.sleep(0.1)
except Exception as e:
logger.error(f"Failed to start probe sniffer: {e}")
def arp_listener() -> None:
"""
ARP listener via netlink RTM_NEWNEIGH.
Kernel pushes neighbor discovery events.
"""
try:
s = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, NETLINK_ROUTE)
s.bind((os.getpid(), RTMGRP_NEIGH))
logger.info("ARP listener started")
while True:
try:
data = s.recv(65535)
parse_netlink(data)
except Exception as e:
logger.debug(f"ARP listener error: {e}")
time.sleep(0.1)
except Exception as e:
logger.error(f"Failed to start ARP listener: {e}")
def parse_netlink(data: bytes) -> None:
"""Parse netlink messages for neighbor events."""
try:
offset = 0
while offset < len(data):
if offset + 16 > len(data):
break
nlmsg_len, nlmsg_type, _, _, _ = struct.unpack_from('=IHHII', data, offset)
if nlmsg_len < 16:
break
# Verify complete message is available before extracting payload
if offset + nlmsg_len > len(data):
return
payload = data[offset + 16:offset + nlmsg_len]
if nlmsg_type in (RTM_NEWNEIGH, RTM_DELNEIGH):
parse_ndmsg(payload, nlmsg_type)
offset += (nlmsg_len + 3) & ~3
except Exception as e:
logger.debug(f"Netlink parse error: {e}")
def parse_ndmsg(data: bytes, msg_type: int) -> None:
"""Parse ndmsg (neighbor discovery message)."""
try:
if len(data) < 12:
return
state = struct.unpack_from('=H', data, 8)[0]
mac = None
ip = None
offset = 12
while offset + 4 <= len(data):
rta_len, rta_type = struct.unpack_from('=HH', data, offset)
if rta_len < 4:
break
# Verify complete RTA attribute is available before extracting
if offset + rta_len > len(data):
return
val = data[offset + 4:offset + rta_len]
if rta_type == NDA_LLADDR and len(val) == 6:
mac = ':'.join(f'{b:02x}' for b in val)
elif rta_type == NDA_DST:
if len(val) == 4:
ip = socket.inet_ntoa(val)
offset += (rta_len + 3) & ~3
if not mac or mac in ('00:00:00:00:00:00', 'ff:ff:ff:ff:ff:ff'):
return
# Ignore infrastructure IPs
with infrastructure_lock:
if ip and ip in infrastructure_ips:
return
if msg_type == RTM_NEWNEIGH and (state & (NUD_REACHABLE | NUD_STALE | NUD_DELAY | NUD_PROBE)):
on_signal(mac, "arp", ARP_BUMP)
except Exception as e:
logger.debug(f"ndmsg parse error: {e}")
def dhcp_sniffer(iface: str) -> None:
"""
DHCP sniffer via AF_PACKET raw socket.
Captures DHCP traffic passively, extracts client MACs.
"""
try:
s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(0x0003))
s.bind((iface, 0))
logger.info(f"DHCP sniffer started on {iface}")
while True:
try:
data, _ = s.recvfrom(65535)
parse_dhcp(data)
except Exception as e:
logger.debug(f"DHCP sniffer error: {e}")
time.sleep(0.1)
except Exception as e:
logger.error(f"Failed to start DHCP sniffer: {e}")
def parse_dhcp(data: bytes) -> None:
"""Parse DHCP packet and extract client MAC."""
try:
if len(data) < 14:
return
# Ethernet frame
eth_type = struct.unpack('!H', data[12:14])[0]
if eth_type != 0x0800: # not IPv4
return
# IP header
if len(data) < 23:
return
proto = data[23]
if proto != 17: # not UDP
return
# UDP ports
if len(data) < 38:
return
src_port = struct.unpack('!H', data[34:36])[0]
dst_port = struct.unpack('!H', data[36:38])[0]
if not (src_port in (67, 68) and dst_port in (67, 68)):
return
# DHCP payload (starts at byte 42)
if len(data) < 42:
return
dhcp = data[42:]
if len(dhcp) < 236:
return
# MAC address (chaddr at offset 28, 6 bytes)
if len(dhcp) < 34:
return
mac_bytes = dhcp[28:34]
mac = ':'.join(f'{b:02x}' for b in mac_bytes)
if mac in ('00:00:00:00:00:00', 'ff:ff:ff:ff:ff:ff'):
return
# Parse DHCP options (start at byte 240)
if len(dhcp) < 240:
return
msg_type = None
i = 240
while i < len(dhcp):
opt = dhcp[i]
if opt == 255:
break
if opt == 0:
i += 1
continue
if i + 1 >= len(dhcp):
break
length = dhcp[i + 1]
if i + 2 + length > len(dhcp):
break
val = dhcp[i + 2:i + 2 + length]
if opt == 53 and length == 1: # DHCP message type
msg_type = val[0]
i += 2 + length
# DISCOVER or REQUEST indicates presence
if msg_type in (1, 3):
on_signal(mac, "dhcp", DHCP_BUMP)
except Exception as e:
logger.debug(f"DHCP parse error: {e}")
def ble_scanner() -> None:
"""
BLE passive scanner.
Gracefully degrades if bleak is not available.
"""
try:
from bleak import BleakScanner
except ImportError:
logger.warning("bleak not available; BLE scanning disabled")
return
async def run_ble_scan() -> None:
"""Run BLE scan in asyncio loop."""
def on_ble_detect(device, adv_data):
"""Callback for BLE device detection."""
if device and device.address:
on_signal(device.address.lower(), "ble", BLE_BUMP)
try:
async with BleakScanner(
detection_callback=on_ble_detect,
scanning_mode="passive"
):
await asyncio.sleep(float('inf'))
except Exception as e:
logger.error(f"BLE scanner error: {e}")
try:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(run_ble_scan())
except Exception as e:
logger.error(f"Failed to start BLE scanner: {e}")
def decay_loop() -> None:
"""
Background thread: apply decay to all devices every 60 seconds.
Prune devices not seen in 120+ minutes.
"""
while True:
try:
time.sleep(60)
with devices_lock:
now = time.time()
to_prune = []
for mac, dev in devices.items():
elapsed_minutes = (now - dev.last_seen) / 60.0
# Apply decay
dev.certainty = dev.certainty * math.exp(-DECAY_LAMBDA * elapsed_minutes)
# Mark for pruning if not seen in 120+ minutes
if elapsed_minutes >= 120:
to_prune.append(mac)
# Prune old devices
for mac in to_prune:
del devices[mac]
logger.debug(f"Pruned device {mac}")
# Update all person states due to decay
for person_id in set(d.person_id for d in devices.values() if d.person_id):
update_person_state(person_id)
except Exception as e:
logger.error(f"Decay loop error: {e}")
def status_handler(request, client_address, server):
"""HTTP request handler for status endpoint."""
class StatusHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/":
try:
with persons_lock:
person_list = [
{
"id": p.person_id,
"name": p.name,
"status": p.status,
"last_change": p.last_change
}
for p in persons.values()
]
response = json.dumps({"persons": person_list})
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(response.encode("utf-8"))
except Exception as e:
logger.error(f"Status handler error: {e}")
self.send_response(500)
self.end_headers()
else:
self.send_response(404)
self.end_headers()
def log_message(self, format, *args):
"""Suppress HTTP server logs."""
pass
return StatusHandler(request, client_address, server)
def status_server() -> None:
"""Start HTTP status server on 127.0.0.1:STATUS_PORT."""
try:
server = http.server.HTTPServer(
("127.0.0.1", STATUS_PORT),
status_handler
)
logger.info(f"Status server started on http://127.0.0.1:{STATUS_PORT}")
server.serve_forever()
except Exception as e:
logger.error(f"Failed to start status server: {e}")
def main() -> None:
"""Initialize and run all sensor threads."""
init_db()
seed_persons_from_db()
threads = [
threading.Thread(target=probe_sniffer, args=(MONITOR_IFACE,), daemon=True, name="probe-sniffer"),
threading.Thread(target=arp_listener, daemon=True, name="arp-listener"),
threading.Thread(target=dhcp_sniffer, args=(DATA_IFACE,), daemon=True, name="dhcp-sniffer"),
threading.Thread(target=ble_scanner, daemon=True, name="ble-scanner"),
threading.Thread(target=decay_loop, daemon=True, name="decay-loop"),
threading.Thread(target=status_server, daemon=True, name="status-server"),
]
for t in threads:
t.start()
logger.info("Sensor daemon started")
# Handle shutdown gracefully
import signal
signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
signal.signal(signal.SIGINT, lambda *_: sys.exit(0))
for t in threads:
t.join()
if __name__ == "__main__":
main()
+33
View File
@@ -0,0 +1,33 @@
CREATE TABLE IF NOT EXISTS persons (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
notes TEXT
);
CREATE TABLE IF NOT EXISTS devices (
mac TEXT PRIMARY KEY,
person_id INTEGER REFERENCES persons(id),
label TEXT,
added_at REAL DEFAULT (unixepoch())
);
CREATE TABLE IF NOT EXISTS signals (
id INTEGER PRIMARY KEY,
mac TEXT NOT NULL,
signal_type TEXT NOT NULL,
certainty REAL NOT NULL,
ts REAL DEFAULT (unixepoch())
);
CREATE TABLE IF NOT EXISTS occupancy_log (
id INTEGER PRIMARY KEY,
person_id INTEGER REFERENCES persons(id),
old_status TEXT,
new_status TEXT NOT NULL,
ts REAL DEFAULT (unixepoch())
);
CREATE INDEX IF NOT EXISTS idx_signals_mac ON signals(mac);
CREATE INDEX IF NOT EXISTS idx_signals_ts ON signals(ts);
CREATE INDEX IF NOT EXISTS idx_occupancy_person ON occupancy_log(person_id);
CREATE INDEX IF NOT EXISTS idx_devices_person ON devices(person_id);