Files

T

n0mad1k 2093b737c3 Make distributable: config.toml support, blanked Matrix defaults, env creds fallback

- ~/.config/coe-mirror/config.toml (tomllib, Python 3.11+ required)
- Precedence uniform for every key: env var > config file > hardcoded default
- API_BASE, OWNERS, PCLOUD_TARGET, PENDING_DIR, matrix homeserver/room resolved
  via _resolve() helper
- Matrix homeserver/room now default to empty; _send_matrix short-circuits when
  either is blank so forks don't spam the original author's room
- _fetch_creds falls back to COE_MIRROR_MATRIX_USER/PASSWORD env vars when the
  Infisical creds binary is unavailable (required for external distribution)
- CLI subcommands: config init (writes example TOML, won't overwrite), config path
  Bare invocation and 'run' still work unchanged for the systemd timer
- README rewritten as a public quick-start with the precedence table and Matrix
  as opt-in
- Tests: 3 new for config precedence, 1 for Matrix silent no-op when blank,
  2 for creds env fallback (26 total, all pass)

DevTrack #1285

2026-07-07 11:04:07 -04:00

4.2 KiB

Raw Blame History

coe-mirror

Mirror repos from a Forgejo/Gitea instance to a local directory (typically pCloud). Built for git.churchofmalware.org but usable against any Forgejo/Gitea host.

What it does

For every owner in your config, it fetches the repo list, downloads a zip of the current HEAD of the default branch, and writes it to <target>/<owner>/<repo>_<sha8>.zip. Old snapshots stay as history — new commits produce new files. Runs unattended via a systemd user timer.

Requirements

Python 3.11+ (uses stdlib tomllib)
pip install -r requirements.txt

Quick start

git clone <this repo>
cd coe-mirror
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python coe_mirror.py config init         # writes ~/.config/coe-mirror/config.toml
$EDITOR ~/.config/coe-mirror/config.toml # edit owners + target dir
python coe_mirror.py run                 # first mirror

Add or remove owners any time by editing the owners = [...] list in the config file.

Configuration

Precedence (uniform for every key): env var > config file > hardcoded default.

Config file location: ~/.config/coe-mirror/config.toml (override with COE_MIRROR_CONFIG=/path/to/file.toml).

Full schema:

api_base      = "https://git.churchofmalware.org"
pcloud_target = "~/pCloudDrive/Hacking/ChurchOfMalware"
owners        = ["Nightmare_Eclipse", "shai_hulud"]
# pending_dir = "~/.local/share/coe-mirror/pending"  # optional

[matrix]  # optional; leave blank to disable notifications
homeserver = ""
room = ""

Env-var overrides (all optional; each beats the config file):

Var	Purpose
`COE_MIRROR_CONFIG`	Alternate path to config.toml
`COE_MIRROR_API_BASE`	Forgejo/Gitea base URL
`COE_MIRROR_OWNERS`	Comma-separated owners (`"a,b,c"`)
`COE_MIRROR_PCLOUD_TARGET`	Target directory
`COE_MIRROR_PENDING_DIR`	Local staging dir
`COE_MIRROR_MATRIX_HOMESERVER`	Matrix homeserver URL
`COE_MIRROR_MATRIX_ROOM`	Matrix room ID
`COE_MIRROR_MATRIX_USER`	Matrix bot user (fallback if `creds` binary unavailable)
`COE_MIRROR_MATRIX_PASSWORD`	Matrix bot password (fallback)

CLI

python coe_mirror.py                # run the mirror (default)
python coe_mirror.py run            # same
python coe_mirror.py config init    # write example config (won't overwrite)
python coe_mirror.py config path    # print config file path
python coe_mirror.py --help

Install as a systemd user timer

bash install.sh

Timer defaults to 00:00 and 12:00 daily with boot catch-up (Persistent=true). Missed windows run on next boot.

Matrix notifications (optional)

Disabled by default. Leave [matrix] blank to skip.
If both homeserver and room are configured, the tool posts a summary after every run that lands at least one new zip (or reports any owner whose fetch failed).
Credential lookup order: creds get COE_MIRROR_BOT_USER matrix / COE_MIRROR_BOT_PASSWORD matrix (Infisical CLI, if you use it), then COE_MIRROR_MATRIX_USER / COE_MIRROR_MATRIX_PASSWORD env vars. Tokens cached at ~/.local/share/coe-mirror/matrix-token.json (mode 0600), auto-refreshed on expiry or 401.
Matrix failures are non-fatal — a warning is logged and the run proceeds.

Filename convention

Final: <target>/<owner>/<repo>_<sha8>.zip
Staging (while pending flush): <pending_dir>/<owner>__<repo>_<sha8>.zip — owner prefix prevents cross-owner collisions when two owners have identically-named repos.

Behavior when the target is unavailable

Downloads land in pending_dir. Next run (or any subsequent invocation) flushes them to the target once it's healthy. Owner-scoped: one owner's fetch failure logs and continues; the next run retries.

Logs

File: ~/.local/share/coe-mirror/coe-mirror.log (rotated at 5 MB, 3 backups)
Journal: journalctl --user -u coe-mirror.service -n 100

Storage

Old snapshots accumulate — no automatic pruning. Prune the target directory manually if you need to reclaim space.

Testing

python -m pytest tests/ -v

Uninstall

systemctl --user disable --now coe-mirror.timer
rm ~/.config/systemd/user/coe-mirror.{service,timer}
systemctl --user daemon-reload

4.2 KiB Raw Blame History