n0mad1k
2093b737c3
- ~/.config/coe-mirror/config.toml (tomllib, Python 3.11+ required) - Precedence uniform for every key: env var > config file > hardcoded default - API_BASE, OWNERS, PCLOUD_TARGET, PENDING_DIR, matrix homeserver/room resolved via _resolve() helper - Matrix homeserver/room now default to empty; _send_matrix short-circuits when either is blank so forks don't spam the original author's room - _fetch_creds falls back to COE_MIRROR_MATRIX_USER/PASSWORD env vars when the Infisical creds binary is unavailable (required for external distribution) - CLI subcommands: config init (writes example TOML, won't overwrite), config path Bare invocation and 'run' still work unchanged for the systemd timer - README rewritten as a public quick-start with the precedence table and Matrix as opt-in - Tests: 3 new for config precedence, 1 for Matrix silent no-op when blank, 2 for creds env fallback (26 total, all pass) DevTrack #1285
118 lines
4.2 KiB
Markdown
118 lines
4.2 KiB
Markdown
# coe-mirror
|
|
|
|
Mirror repos from a Forgejo/Gitea instance to a local directory (typically pCloud). Built for `git.churchofmalware.org` but usable against any Forgejo/Gitea host.
|
|
|
|
## What it does
|
|
|
|
For every owner in your config, it fetches the repo list, downloads a zip of the current HEAD of the default branch, and writes it to `<target>/<owner>/<repo>_<sha8>.zip`. Old snapshots stay as history — new commits produce new files. Runs unattended via a systemd user timer.
|
|
|
|
## Requirements
|
|
|
|
- Python 3.11+ (uses stdlib `tomllib`)
|
|
- `pip install -r requirements.txt`
|
|
|
|
## Quick start
|
|
|
|
```
|
|
git clone <this repo>
|
|
cd coe-mirror
|
|
python3 -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
|
|
python coe_mirror.py config init # writes ~/.config/coe-mirror/config.toml
|
|
$EDITOR ~/.config/coe-mirror/config.toml # edit owners + target dir
|
|
python coe_mirror.py run # first mirror
|
|
```
|
|
|
|
Add or remove owners any time by editing the `owners = [...]` list in the config file.
|
|
|
|
## Configuration
|
|
|
|
**Precedence** (uniform for every key): env var > config file > hardcoded default.
|
|
|
|
**Config file** location: `~/.config/coe-mirror/config.toml` (override with `COE_MIRROR_CONFIG=/path/to/file.toml`).
|
|
|
|
Full schema:
|
|
|
|
```toml
|
|
api_base = "https://git.churchofmalware.org"
|
|
pcloud_target = "~/pCloudDrive/Hacking/ChurchOfMalware"
|
|
owners = ["Nightmare_Eclipse", "shai_hulud"]
|
|
# pending_dir = "~/.local/share/coe-mirror/pending" # optional
|
|
|
|
[matrix] # optional; leave blank to disable notifications
|
|
homeserver = ""
|
|
room = ""
|
|
```
|
|
|
|
**Env-var overrides** (all optional; each beats the config file):
|
|
|
|
| Var | Purpose |
|
|
|---|---|
|
|
| `COE_MIRROR_CONFIG` | Alternate path to config.toml |
|
|
| `COE_MIRROR_API_BASE` | Forgejo/Gitea base URL |
|
|
| `COE_MIRROR_OWNERS` | Comma-separated owners (`"a,b,c"`) |
|
|
| `COE_MIRROR_PCLOUD_TARGET` | Target directory |
|
|
| `COE_MIRROR_PENDING_DIR` | Local staging dir |
|
|
| `COE_MIRROR_MATRIX_HOMESERVER` | Matrix homeserver URL |
|
|
| `COE_MIRROR_MATRIX_ROOM` | Matrix room ID |
|
|
| `COE_MIRROR_MATRIX_USER` | Matrix bot user (fallback if `creds` binary unavailable) |
|
|
| `COE_MIRROR_MATRIX_PASSWORD` | Matrix bot password (fallback) |
|
|
|
|
## CLI
|
|
|
|
```
|
|
python coe_mirror.py # run the mirror (default)
|
|
python coe_mirror.py run # same
|
|
python coe_mirror.py config init # write example config (won't overwrite)
|
|
python coe_mirror.py config path # print config file path
|
|
python coe_mirror.py --help
|
|
```
|
|
|
|
## Install as a systemd user timer
|
|
|
|
```
|
|
bash install.sh
|
|
```
|
|
|
|
Timer defaults to 00:00 and 12:00 daily with boot catch-up (`Persistent=true`). Missed windows run on next boot.
|
|
|
|
## Matrix notifications (optional)
|
|
|
|
- **Disabled by default.** Leave `[matrix]` blank to skip.
|
|
- If both `homeserver` and `room` are configured, the tool posts a summary after every run that lands at least one new zip (or reports any owner whose fetch failed).
|
|
- Credential lookup order: `creds get COE_MIRROR_BOT_USER matrix` / `COE_MIRROR_BOT_PASSWORD matrix` (Infisical CLI, if you use it), then `COE_MIRROR_MATRIX_USER` / `COE_MIRROR_MATRIX_PASSWORD` env vars. Tokens cached at `~/.local/share/coe-mirror/matrix-token.json` (mode 0600), auto-refreshed on expiry or 401.
|
|
- Matrix failures are non-fatal — a warning is logged and the run proceeds.
|
|
|
|
## Filename convention
|
|
|
|
- Final: `<target>/<owner>/<repo>_<sha8>.zip`
|
|
- Staging (while pending flush): `<pending_dir>/<owner>__<repo>_<sha8>.zip` — owner prefix prevents cross-owner collisions when two owners have identically-named repos.
|
|
|
|
## Behavior when the target is unavailable
|
|
|
|
Downloads land in `pending_dir`. Next run (or any subsequent invocation) flushes them to the target once it's healthy. Owner-scoped: one owner's fetch failure logs and continues; the next run retries.
|
|
|
|
## Logs
|
|
|
|
- File: `~/.local/share/coe-mirror/coe-mirror.log` (rotated at 5 MB, 3 backups)
|
|
- Journal: `journalctl --user -u coe-mirror.service -n 100`
|
|
|
|
## Storage
|
|
|
|
Old snapshots accumulate — no automatic pruning. Prune the target directory manually if you need to reclaim space.
|
|
|
|
## Testing
|
|
|
|
```
|
|
python -m pytest tests/ -v
|
|
```
|
|
|
|
## Uninstall
|
|
|
|
```
|
|
systemctl --user disable --now coe-mirror.timer
|
|
rm ~/.config/systemd/user/coe-mirror.{service,timer}
|
|
systemctl --user daemon-reload
|
|
```
|