mosaic

T

n0mad1k 943baa3c02 Add ingest command for local file and dump import

New 'mosaic ingest' CLI command bypasses HTTP collection to import
raw documents (PDF, HTML, text) or pre-extracted JSONL data directly
into the pipeline.

Modes:
- Single/multiple files: -f file1.pdf -f file2.html -s source_name
- Directory bulk import: -d /path/to/dump/ -s source_name -r
- JSONL extracted data: -f data.jsonl -s name -c tools
- Real-time watch: -d /incoming/ -s name -w (polls every 2s)
- Auto-parse: --parse flag runs parser on newly ingested docs

Files are hashed, deduplicated, and stored in the cache using the
same hash-sharded scheme as HTTP-collected documents.

DevTrack: mosaic #498

2026-04-10 06:57:54 -04:00

analysis

Add field manual generator — produces structured Markdown operator reference from extracted intel

2026-03-19 12:46:36 -04:00

collectors

Add source profiles for physical tradecraft: CIA RDP, FAS/IRP, Army pubs, Cryptome, Grugq

2026-03-19 10:22:02 -04:00

data

Initial scaffold: .gitignore, config, requirements, data seeds, directory structure

2026-03-19 08:34:32 -04:00

extractors

Fix deprecated datetime.utcnow() and test assertion for clean_text