n0mad1k/coe-mirror
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make distributable: config.toml support, blanked Matrix defaults, env creds fallback

Browse Source 
- ~/.config/coe-mirror/config.toml (tomllib, Python 3.11+ required)
- Precedence uniform for every key: env var > config file > hardcoded default
- API_BASE, OWNERS, PCLOUD_TARGET, PENDING_DIR, matrix homeserver/room resolved
  via _resolve() helper
- Matrix homeserver/room now default to empty; _send_matrix short-circuits when
  either is blank so forks don't spam the original author's room
- _fetch_creds falls back to COE_MIRROR_MATRIX_USER/PASSWORD env vars when the
  Infisical creds binary is unavailable (required for external distribution)
- CLI subcommands: config init (writes example TOML, won't overwrite), config path
  Bare invocation and 'run' still work unchanged for the systemd timer
- README rewritten as a public quick-start with the precedence table and Matrix
  as opt-in
- Tests: 3 new for config precedence, 1 for Matrix silent no-op when blank,
  2 for creds env fallback (26 total, all pass)

DevTrack #1285
This commit is contained in:
n0mad1k
2026-07-07 11:04:07 -04:00
parent 9ffe38cbbc
commit 376115a9de
4 changed files with 357 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+83 -38
View File
@@ -1,66 +1,111 @@
# coe-mirror
Mirror Nightmare_Eclipse repos from Church of Malware to pCloud.
Mirror repos from a Forgejo/Gitea instance to a local directory (typically pCloud). Built for `git.churchofmalware.org` but usable against any Forgejo/Gitea host.
## What it does
Monitors `Nightmare_Eclipse` on `git.churchofmalware.org` for new or updated repos. Downloads per-commit zip snapshots to `~/pCloudDrive/Hacking/Nightmare_Eclipse/`. Falls back to local staging when pCloud is down. Runs at 00:00 and 12:00 daily with boot catch-up.
For every owner in your config, it fetches the repo list, downloads a zip of the current HEAD of the default branch, and writes it to `<target>/<owner>/<repo>_<sha8>.zip`. Old snapshots stay as history — new commits produce new files. Runs unattended via a systemd user timer.
## Filename convention
## Requirements
`{repo}_{short_sha}.zip` where short_sha is the first 8 chars of the HEAD commit SHA on the default branch. New commits produce new files; old snapshots stay as history.
- Python 3.11+ (uses stdlib `tomllib`)
- `pip install -r requirements.txt`
## Install
## Quick start
```
git clone <this repo>
cd coe-mirror
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python coe_mirror.py config init # writes ~/.config/coe-mirror/config.toml
$EDITOR ~/.config/coe-mirror/config.toml # edit owners + target dir
python coe_mirror.py run # first mirror
```
Add or remove owners any time by editing the `owners = [...]` list in the config file.
## Configuration
**Precedence** (uniform for every key): env var > config file > hardcoded default.
**Config file** location: `~/.config/coe-mirror/config.toml` (override with `COE_MIRROR_CONFIG=/path/to/file.toml`).
Full schema:
```toml
api_base = "https://git.churchofmalware.org"
pcloud_target = "~/pCloudDrive/Hacking/ChurchOfMalware"
owners = ["Nightmare_Eclipse", "shai_hulud"]
# pending_dir = "~/.local/share/coe-mirror/pending" # optional
[matrix] # optional; leave blank to disable notifications
homeserver = ""
room = ""
```
**Env-var overrides** (all optional; each beats the config file):
| Var | Purpose |
|---|---|
| `COE_MIRROR_CONFIG` | Alternate path to config.toml |
| `COE_MIRROR_API_BASE` | Forgejo/Gitea base URL |
| `COE_MIRROR_OWNERS` | Comma-separated owners (`"a,b,c"`) |
| `COE_MIRROR_PCLOUD_TARGET` | Target directory |
| `COE_MIRROR_PENDING_DIR` | Local staging dir |
| `COE_MIRROR_MATRIX_HOMESERVER` | Matrix homeserver URL |
| `COE_MIRROR_MATRIX_ROOM` | Matrix room ID |
| `COE_MIRROR_MATRIX_USER` | Matrix bot user (fallback if `creds` binary unavailable) |
| `COE_MIRROR_MATRIX_PASSWORD` | Matrix bot password (fallback) |
## CLI
```
python coe_mirror.py # run the mirror (default)
python coe_mirror.py run # same
python coe_mirror.py config init # write example config (won't overwrite)
python coe_mirror.py config path # print config file path
python coe_mirror.py --help
```
## Install as a systemd user timer
```
cd ~/tools/coe-mirror
bash install.sh
```
## Manual run
Timer defaults to 00:00 and 12:00 daily with boot catch-up (`Persistent=true`). Missed windows run on next boot.
```
systemctl --user start coe-mirror.service
# or directly:
python3 ~/tools/coe-mirror/coe_mirror.py
```
## Matrix notifications (optional)
- **Disabled by default.** Leave `[matrix]` blank to skip.
- If both `homeserver` and `room` are configured, the tool posts a summary after every run that lands at least one new zip (or reports any owner whose fetch failed).
- Credential lookup order: `creds get COE_MIRROR_BOT_USER matrix` / `COE_MIRROR_BOT_PASSWORD matrix` (Infisical CLI, if you use it), then `COE_MIRROR_MATRIX_USER` / `COE_MIRROR_MATRIX_PASSWORD` env vars. Tokens cached at `~/.local/share/coe-mirror/matrix-token.json` (mode 0600), auto-refreshed on expiry or 401.
- Matrix failures are non-fatal — a warning is logged and the run proceeds.
## Filename convention
- Final: `<target>/<owner>/<repo>_<sha8>.zip`
- Staging (while pending flush): `<pending_dir>/<owner>__<repo>_<sha8>.zip` — owner prefix prevents cross-owner collisions when two owners have identically-named repos.
## Behavior when the target is unavailable
Downloads land in `pending_dir`. Next run (or any subsequent invocation) flushes them to the target once it's healthy. Owner-scoped: one owner's fetch failure logs and continues; the next run retries.
## Logs
- File: `~/.local/share/coe-mirror/coe-mirror.log` (rotated at 5 MB, 3 backups)
- Journal: `journalctl --user -u coe-mirror.service -n 100`
## Storage estimate
## Storage
8 repos × 2 runs/day × ~1 commit/repo/day ≈ 16 zips/day ≈ ~5 MB/month at current zip sizes (~265 KB each). Old snapshots accumulate — no automatic pruning. Manually prune `~/pCloudDrive/Hacking/Nightmare_Eclipse/` if needed.
## Behavior when pCloud is down
Downloads land in `~/.local/share/coe-mirror/pending/`. Next run (or any subsequent invocation) flushes them to pCloud once the mount is healthy. No auto-start of the pCloud daemon — by design, because the daemon is a GUI AppImage with no systemd user unit. If pCloud is offline, ensure the user session is active and pCloud is running before the next scheduled run, or trigger manually.
## Catch-up after machine was off
The timer is `Persistent=true` and `OnBootSec=2min`. Missed windows run automatically on next boot. The tool also diffs against the pCloud directory, so even if a run is skipped entirely, the next run catches up everything that's missing.
## Matrix notification
After every run that lands at least one new zip, coe-mirror posts a summary
to the General room (`!REDACTEDROOM:m.example.org`) on
`m.example.org`.
- Authentication uses `MATRIX_USER` + `MATRIX_PASSWORD` from Infisical folder
`matrix`. Tokens are cached at `~/.local/share/coe-mirror/matrix-token.json`
(mode 0600) and auto-refreshed on expiry or 401.
- Failures are non-fatal: a Matrix outage logs a warning and the rest of the
run proceeds normally.
- Override the target room for testing: `COE_MIRROR_MATRIX_ROOM=! ...:host`.
- Override the homeserver: `COE_MIRROR_MATRIX_HOMESERVER=https://...`.
Old snapshots accumulate — no automatic pruning. Prune the target directory manually if you need to reclaim space.
## Testing
```
cd ~/tools/coe-mirror
python3 -m pytest tests/ -v
python -m pytest tests/ -v
```
## Uninstall