943baa3c02
Add ingest command for local file and dump import
...
New 'mosaic ingest' CLI command bypasses HTTP collection to import
raw documents (PDF, HTML, text) or pre-extracted JSONL data directly
into the pipeline.
Modes:
- Single/multiple files: -f file1.pdf -f file2.html -s source_name
- Directory bulk import: -d /path/to/dump/ -s source_name -r
- JSONL extracted data: -f data.jsonl -s name -c tools
- Real-time watch: -d /incoming/ -s name -w (polls every 2s)
- Auto-parse: --parse flag runs parser on newly ingested docs
Files are hashed, deduplicated, and stored in the cache using the
same hash-sharded scheme as HTTP-collected documents.
DevTrack: mosaic #498
2026-04-10 06:57:54 -04:00
58c3b75e5e
Add OPSEC Principles, Physical Access, and Cyber Implants manuals
2026-03-19 17:15:43 -04:00
0811e04cd2
Add 4 operator manuals: HUMINT, Surveillance, Covert Comms, Cover & Identity
2026-03-19 17:07:15 -04:00
d2e27d8357
Add field manual generator — produces structured Markdown operator reference from extracted intel
2026-03-19 12:46:36 -04:00
7cbc79c229
Add source profiles for physical tradecraft: CIA RDP, FAS/IRP, Army pubs, Cryptome, Grugq
2026-03-19 10:22:02 -04:00
b67780c552
Fix crawler scoping: BFS stays within source URL prefix, only collect pattern-matched URLs
2026-03-19 09:59:58 -04:00
9ab6c04884
Wire up collect and parse CLI commands to actual engines
2026-03-19 09:44:53 -04:00
5d325efc75
Fix Click deprecation warning, use standard invoke_without_command pattern
2026-03-19 09:03:41 -04:00
0cfd7dc32d
Fix deprecated datetime.utcnow() and test assertion for clean_text
2026-03-19 08:56:08 -04:00
c79f81d9c6
Add analysis engine: batch pipeline, interactive Q&A, context builder
2026-03-19 08:53:27 -04:00
10435118a9
Add core test suite for storage, utils, models, and parsers
2026-03-19 08:53:12 -04:00
ca53524118
Add extractors: TTP, tool, tradecraft, surveillance, infrastructure, entity, MITRE mapper + prompts
2026-03-19 08:51:37 -04:00
41cbdf5b04
Add parser engine: base with subprocess isolation, HTML/PDF/email/cable/text parsers
2026-03-19 08:47:12 -04:00
4f381436a4
Add collector engine: base, custom source, profile loader, wikileaks helpers
2026-03-19 08:39:17 -04:00
176156d4e9
Add main CLI with Click subcommands and interactive menu
2026-03-19 08:39:06 -04:00
aa0f35803f
Initial scaffold: .gitignore, config, requirements, data seeds, directory structure
2026-03-19 08:34:32 -04:00
n0mad1k