# mosaic

Intelligence extraction platform for large open-source and declassified document corpora.
Mosaic collects public leak/disclosure archives (WikiLeaks, Cryptome, FAS/IRP, CIA CREST/RDP,
Army field manuals, and similar), parses heterogeneous documents, extracts structured
intelligence (entities, tooling, TTPs, infrastructure, tradecraft), and lets you query and
synthesize the results — all locally, in SQLite.

> Everything Mosaic ingests is publicly available or previously declassified material.
> It ships with no corpus and no data — you point it at the sources you are authorized to use.

## Pipeline

```
collect  →  parse  →  extract  →  analyze  →  query
```

- **collect** — source-profile-driven crawlers (rate-limited, resumable) into a hash-sharded store
- **parse** — subprocess-isolated PDF / HTML / email / cable / text parsers with memory + time caps
- **extract** — Claude-backed extractors: entities, tools, MITRE ATT&CK mapping, infrastructure,
  surveillance, tradecraft, TTPs
- **analyze** — BM25/FTS5 context builder, batch + interactive analysis, field-manual generation
- **query** — full-text search (SQLite FTS5) over everything extracted

## Install

```bash
git clone https://git.churchofmalware.org/n0mad1k/mosaic.git
cd mosaic
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
```

## Configure

Copy the example config and set your options:

```bash
cp config/mosaic.conf.example config/mosaic.conf   # if present; otherwise mosaic writes defaults
```

The Anthropic API key is read from the `ANTHROPIC_API_KEY` environment variable (preferred):

```bash
export ANTHROPIC_API_KEY=sk-ant-...
```

A commented `api_key` fallback exists in `config/mosaic.conf`, but the env var takes priority and
is the recommended path. No key is required for `collect`, `parse`, or `query` — only for the
Claude-backed `extract`/`analyze` stages.

## Usage

```bash
python3 mosaic.py                 # interactive Rich menu
python3 mosaic.py --help          # full CLI

# typical flow
python3 mosaic.py collect --profile collectors/profiles/cryptome.yaml
python3 mosaic.py parse
python3 mosaic.py extract --dry-run     # estimate token cost first
python3 mosaic.py extract
python3 mosaic.py query "kerberos delegation"
```

Source profiles live in `collectors/profiles/`. Copy `example.yaml` to add your own.

## Field manuals

`output/manuals/` contains reference manuals synthesized from public and declassified source
material (HUMINT, surveillance/counter-surveillance, covert communications, physical access,
cover & identity, OPSEC, and cyber implants). They are generated artifacts produced by the
`analyze` stage and are included as worked examples.

## Tests

```bash
pytest tests/
```

## License

PolyForm Noncommercial 1.0.0 — see [LICENSE](LICENSE). Commercial licensing on request.