mosaic

Intelligence extraction platform for large open-source and declassified document corpora. Mosaic collects public leak/disclosure archives (WikiLeaks, Cryptome, FAS/IRP, CIA CREST/RDP, Army field manuals, and similar), parses heterogeneous documents, extracts structured intelligence (entities, tooling, TTPs, infrastructure, tradecraft), and lets you query and synthesize the results — all locally, in SQLite.

Everything Mosaic ingests is publicly available or previously declassified material. It ships with no corpus and no data — you point it at the sources you are authorized to use.

Pipeline

collect  →  parse  →  extract  →  analyze  →  query

collect — source-profile-driven crawlers (rate-limited, resumable) into a hash-sharded store
parse — subprocess-isolated PDF / HTML / email / cable / text parsers with memory + time caps
extract — Claude-backed extractors: entities, tools, MITRE ATT&CK mapping, infrastructure, surveillance, tradecraft, TTPs
analyze — BM25/FTS5 context builder, batch + interactive analysis, field-manual generation
query — full-text search (SQLite FTS5) over everything extracted

Install

git clone https://git.churchofmalware.org/n0mad1k/mosaic.git
cd mosaic
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Configure

Copy the example config and set your options:

cp config/mosaic.conf.example config/mosaic.conf   # if present; otherwise mosaic writes defaults

The Anthropic API key is read from the ANTHROPIC_API_KEY environment variable (preferred):

export ANTHROPIC_API_KEY=sk-ant-...

A commented api_key fallback exists in config/mosaic.conf, but the env var takes priority and is the recommended path. No key is required for collect, parse, or query — only for the Claude-backed extract/analyze stages.

Usage

python3 mosaic.py                 # interactive Rich menu
python3 mosaic.py --help          # full CLI

# typical flow
python3 mosaic.py collect --profile collectors/profiles/cryptome.yaml
python3 mosaic.py parse
python3 mosaic.py extract --dry-run     # estimate token cost first
python3 mosaic.py extract
python3 mosaic.py query "kerberos delegation"

Source profiles live in collectors/profiles/. Copy example.yaml to add your own.

Field manuals

output/manuals/ contains reference manuals synthesized from public and declassified source material (HUMINT, surveillance/counter-surveillance, covert communications, physical access, cover & identity, OPSEC, and cyber implants). They are generated artifacts produced by the analyze stage and are included as worked examples.

Tests

pytest tests/

License

PolyForm Noncommercial 1.0.0 — see LICENSE. Commercial licensing on request.

2.8 KiB Raw Blame History