n0mad1k/bigbrother
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PCAP max compression: zstd -19 on rotation for storage and transfer efficiency

Browse Source
This commit is contained in:
n0mad1k
2026-03-17 11:01:19 -04:00
parent d3d1deb99c
commit 3041943c21
1 changed files with 16 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BIGBROTHER_DESIGN.md
+16 -7
View File
@@ -246,13 +246,20 @@ Scope is the operator's responsibility. An optional `scope.yaml` can be loaded t
- Promiscuous mode on bridge or tap interface - Promiscuous mode on bridge or tap interface
- BPF filters: full capture, credentials-only, DNS-only, or custom - BPF filters: full capture, credentials-only, DNS-only, or custom
- PCAP rotation: size-based (100MB default) or time-based (1h default) - PCAP rotation: size-based (100MB uncompressed default) or time-based (1h default)
- AES-256-GCM encryption before writing to disk - **Max compression pipeline on rotation**: zstd level 19 (max) → AES-256-GCM encrypt → .pcap.zst.enc
- zstd -19 achieves 10-15x compression on typical network PCAPs (100MB → 7-10MB)
- On 128GB SD: ~1.2TB effective PCAP storage (days of full capture on busy networks)
- Compressed files dramatically faster to exfil when operator pulls data over Tailscale/WG
- Decompression on operator's workstation: `zstd -d` then open in Wireshark
- CPU cost: zstd -19 on Pi 4 compresses at ~5-10MB/s (runs as background task after rotation, not inline)
- Pi Zero: use zstd -3 (fast mode, still 5-8x compression, minimal CPU)
- Snap length configurable: 96 bytes (headers only) for Pi Zero, 65535 (full) for Pi 4+ - Snap length configurable: 96 bytes (headers only) for Pi Zero, 65535 (full) for Pi 4+
- Disk usage monitoring — auto-purge oldest PCAPs when threshold hit - Disk usage monitoring — auto-purge oldest PCAPs when threshold hit
- PcapNG format support for metadata embedding (interface info, comments, timestamps)
**Resources**: RAM 30-80MB (snap length dependent), CPU 5-15%, disk varies **Resources**: RAM 30-80MB (snap length dependent), CPU 5-15% capture + compression spikes on rotation, disk varies
**Dependencies**: libpcap, scapy **Dependencies**: libpcap, scapy, zstandard (Python binding)
--- ---
@@ -1450,7 +1457,9 @@ security:
capture: capture:
pcap_rotation_mb: 100 pcap_rotation_mb: 100
pcap_rotation_hours: 1 pcap_rotation_hours: 1
pcap_compression: "zstd" # zstd compression on rotation (local CPU, zero network signature) pcap_compression: "zstd" # zstd max compression on rotation
pcap_compression_level: 19 # 1-22, default 19 (max practical). Pi Zero: auto-downgrade to 3
pcap_format: "pcapng" # pcapng (preferred) or pcap
snap_length: "auto" # auto (by hardware tier) | 96 | 65535 snap_length: "auto" # auto (by hardware tier) | 96 | 65535
bpf_filter: "full_capture" # full_capture | credentials | dns | custom bpf_filter: "full_capture" # full_capture | credentials | dns | custom
custom_bpf: "" custom_bpf: ""
@@ -1561,7 +1570,7 @@ When enabled: ARP spoof validates each target IP, DHCP spoof only responds to in
## 10. Hardware Tiers ## 10. Hardware Tiers
**Minimum SD card**: 128GB (high-endurance industrial). Provides ample PCAP storage with zstd compression on rotation. **Minimum SD card**: 128GB (high-endurance industrial). With zstd -19 compression (10-15x on PCAPs), 128GB provides ~1.2TB effective PCAP storage. Compressed PCAPs also transfer 10-15x faster when operator pulls data over C2 channel.
| Capability | Pi Zero 2W (512MB) | Pi 4 (4GB) | Debian Host | | Capability | Pi Zero 2W (512MB) | Pi 4 (4GB) | Debian Host |
|---|:---:|:---:|:---:| |---|:---:|:---:|:---:|
@@ -1569,7 +1578,7 @@ When enabled: ARP spoof validates each target IP, DHCP spoof only responds to in
| **Max active modules** | 1 | 6 | Unlimited | | **Max active modules** | 1 | 6 | Unlimited |
| **Snap length** | 96 bytes | 65535 | 65535 | | **Snap length** | 96 bytes | 65535 | 65535 |
| **Flow table max** | 5,000 | 50,000 | 100,000 | | **Flow table max** | 5,000 | 50,000 | 100,000 |
| **PCAP compression** | zstd | zstd | zstd | | **PCAP compression** | zstd -3 (fast) | zstd -19 (max) | zstd -19 (max) |
| **Key derivation** | pbkdf2 | argon2id | argon2id | | **Key derivation** | pbkdf2 | argon2id | argon2id |
| **mitmproxy** | No | Yes | Yes | | **mitmproxy** | No | Yes | Yes |
| **file_extractor** | No | Yes | Yes | | **file_extractor** | No | Yes | Yes |