n0mad1k
84 lines
2.8 KiB
Markdown
84 lines
2.8 KiB
Markdown
# mosaic
|
|
|
|
Intelligence extraction platform for large open-source and declassified document corpora.
|
|
Mosaic collects public leak/disclosure archives (WikiLeaks, Cryptome, FAS/IRP, CIA CREST/RDP,
|
|
Army field manuals, and similar), parses heterogeneous documents, extracts structured
|
|
intelligence (entities, tooling, TTPs, infrastructure, tradecraft), and lets you query and
|
|
synthesize the results — all locally, in SQLite.
|
|
|
|
> Everything Mosaic ingests is publicly available or previously declassified material.
|
|
> It ships with no corpus and no data — you point it at the sources you are authorized to use.
|
|
|
|
## Pipeline
|
|
|
|
```
|
|
collect → parse → extract → analyze → query
|
|
```
|
|
|
|
- **collect** — source-profile-driven crawlers (rate-limited, resumable) into a hash-sharded store
|
|
- **parse** — subprocess-isolated PDF / HTML / email / cable / text parsers with memory + time caps
|
|
- **extract** — Claude-backed extractors: entities, tools, MITRE ATT&CK mapping, infrastructure,
|
|
surveillance, tradecraft, TTPs
|
|
- **analyze** — BM25/FTS5 context builder, batch + interactive analysis, field-manual generation
|
|
- **query** — full-text search (SQLite FTS5) over everything extracted
|
|
|
|
## Install
|
|
|
|
```bash
|
|
git clone https://git.churchofmalware.org/n0mad1k/mosaic.git
|
|
cd mosaic
|
|
python3 -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Configure
|
|
|
|
Copy the example config and set your options:
|
|
|
|
```bash
|
|
cp config/mosaic.conf.example config/mosaic.conf # if present; otherwise mosaic writes defaults
|
|
```
|
|
|
|
The Anthropic API key is read from the `ANTHROPIC_API_KEY` environment variable (preferred):
|
|
|
|
```bash
|
|
export ANTHROPIC_API_KEY=sk-ant-...
|
|
```
|
|
|
|
A commented `api_key` fallback exists in `config/mosaic.conf`, but the env var takes priority and
|
|
is the recommended path. No key is required for `collect`, `parse`, or `query` — only for the
|
|
Claude-backed `extract`/`analyze` stages.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python3 mosaic.py # interactive Rich menu
|
|
python3 mosaic.py --help # full CLI
|
|
|
|
# typical flow
|
|
python3 mosaic.py collect --profile collectors/profiles/cryptome.yaml
|
|
python3 mosaic.py parse
|
|
python3 mosaic.py extract --dry-run # estimate token cost first
|
|
python3 mosaic.py extract
|
|
python3 mosaic.py query "kerberos delegation"
|
|
```
|
|
|
|
Source profiles live in `collectors/profiles/`. Copy `example.yaml` to add your own.
|
|
|
|
## Field manuals
|
|
|
|
`output/manuals/` contains reference manuals synthesized from public and declassified source
|
|
material (HUMINT, surveillance/counter-surveillance, covert communications, physical access,
|
|
cover & identity, OPSEC, and cyber implants). They are generated artifacts produced by the
|
|
`analyze` stage and are included as worked examples.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pytest tests/
|
|
```
|
|
|
|
## License
|
|
|
|
PolyForm Noncommercial 1.0.0 — see [LICENSE](LICENSE). Commercial licensing on request.
|