Add README, PolyForm NC license, and harden .gitignore for public release

2026-07-08 10:23:58 -04:00
parent 98dc356645
commit 5b6c5c92eb
3 changed files with 173 additions and 0 deletions
@@ -0,0 +1,83 @@
+# mosaic
+
+Intelligence extraction platform for large open-source and declassified document corpora.
+Mosaic collects public leak/disclosure archives (WikiLeaks, Cryptome, FAS/IRP, CIA CREST/RDP,
+Army field manuals, and similar), parses heterogeneous documents, extracts structured
+intelligence (entities, tooling, TTPs, infrastructure, tradecraft), and lets you query and
+synthesize the results — all locally, in SQLite.
+
+> Everything Mosaic ingests is publicly available or previously declassified material.
+> It ships with no corpus and no data — you point it at the sources you are authorized to use.
+
+## Pipeline
+
+```
+collect  →  parse  →  extract  →  analyze  →  query
+```
+
+- **collect** — source-profile-driven crawlers (rate-limited, resumable) into a hash-sharded store
+- **parse** — subprocess-isolated PDF / HTML / email / cable / text parsers with memory + time caps
+- **extract** — Claude-backed extractors: entities, tools, MITRE ATT&CK mapping, infrastructure,
+  surveillance, tradecraft, TTPs
+- **analyze** — BM25/FTS5 context builder, batch + interactive analysis, field-manual generation
+- **query** — full-text search (SQLite FTS5) over everything extracted
+
+## Install
+
+```bash
+git clone https://git.churchofmalware.org/n0mad1k/mosaic.git
+cd mosaic
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Configure
+
+Copy the example config and set your options:
+
+```bash
+cp config/mosaic.conf.example config/mosaic.conf   # if present; otherwise mosaic writes defaults
+```
+
+The Anthropic API key is read from the `ANTHROPIC_API_KEY` environment variable (preferred):
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+A commented `api_key` fallback exists in `config/mosaic.conf`, but the env var takes priority and
+is the recommended path. No key is required for `collect`, `parse`, or `query` — only for the
+Claude-backed `extract`/`analyze` stages.
+
+## Usage
+
+```bash
+python3 mosaic.py                 # interactive Rich menu
+python3 mosaic.py --help          # full CLI
+
+# typical flow
+python3 mosaic.py collect --profile collectors/profiles/cryptome.yaml
+python3 mosaic.py parse
+python3 mosaic.py extract --dry-run     # estimate token cost first
+python3 mosaic.py extract
+python3 mosaic.py query "kerberos delegation"
+```
+
+Source profiles live in `collectors/profiles/`. Copy `example.yaml` to add your own.
+
+## Field manuals
+
+`output/manuals/` contains reference manuals synthesized from public and declassified source
+material (HUMINT, surveillance/counter-surveillance, covert communications, physical access,
+cover & identity, OPSEC, and cyber implants). They are generated artifacts produced by the
+`analyze` stage and are included as worked examples.
+
+## Tests
+
+```bash
+pytest tests/
+```
+
+## License
+
+PolyForm Noncommercial 1.0.0 — see [LICENSE](LICENSE). Commercial licensing on request.