Background jobs
The daily jobs inside the waterflow container: purge, backup, exchange rates.
app/launch.py starts the daily scheduler (backend/app/jobs/scheduler.py)
next to the API in the same container. Every 30 seconds it reads the schedule from the database and runs each job
that’s due, in this order, each as its own process so one failing doesn’t stop the others. Each logs in with its own
database role and can do only its own job. Output goes to the container’s log (docker compose logs waterflow); each
run’s result (done or failed, when) and its last line of output also show on Server admin.
| Job | Module | Login | Does |
|---|---|---|---|
| purge | app.jobs.account_deletion |
purge_login (account_purger) |
Purges accounts closed more than 30 days ago; deletes sessions past their lifetime (30 days unused, or the user’s own sign-out after inactivity) and old invites and reset links |
| backup | app.jobs.backup |
the database owner | pg_dump of the app (schemas public and waterflow) into /backups; deletes dumps older than about 5.5 days |
| rates | app.jobs.fx_update |
fx_login (fx_updater) |
open.er-api.com (no key), currency-api if that fails, or Open Exchange Rates with OXR_APP_ID; writes fx_rates; safe to re-run |
The launcher also takes a backup before an update applies new migrations.
On Google Cloud Run the server sleeps between visits and gets no CPU then, so it runs with RUN_JOBS=off and
a Cloud Run job runs python -m app.launch jobs every hour, started by Cloud Scheduler. That command looks at the
schedule once, runs what’s due, the same way, and exits (non-zero if a job failed). So each job runs within the
hour after its time. Setup: Run it on Google Cloud Run.
When they run
- Each job runs once a day at its own time: the one the admin sets on Server admin, Daily jobs, in their
own timezone (half-hour steps), or
JOBS_HOUR(UTC, default 19 = 02:00 in Vietnam) until then. A time is kept with its timezone, so daylight saving time doesn’t move it. - A job is due once its time has come since it last started and since its schedule last changed. So a server that was off at that time runs it when it starts again, a new server runs every job at its first start, and moving a time doesn’t run the job at once. A change applies within half a minute, without a restart. A job that failed runs again an hour after it started, until it works (so rates that couldn’t be fetched at the first start arrive once the network allows).
- Only the exchange rates can be turned off (rates then stay as they are): a server can’t do without backups or the purge.
- The schedule and the last results are the
jobstable, reached only through functions: the admin’s (admin_jobs(),admin_set_job(), via/v1/admin/jobs) and the scheduler’s (job_schedule(),job_started(),job_finished()), which it calls aspurge_login. When a job is due: backend/app/core/jobs.py.
Account purge, in short
- A user deletes their account: it closes at once, restorable for 30 days.
- After 30 days the job deletes books only they used, keeps shared books (handing ownership to someone who stays), reduces their user row to a bare id, deletes their password, two-factor secret, recovery codes and sessions, and frees their username.
- It writes purged ids to
PURGE_LOG(/backups/purged_accounts.jsonl, outside the database) before committing. - After restoring a backup:
docker compose exec waterflow python -m app.admin repurgepurges those ids again.
Backups
Owner-only files in the backups volume, about 6 days kept. They’re on the same disk as the server, so
Backups and restore tells operators to copy them elsewhere. Restore steps: that page, and the docstring of
backend/app/jobs/backup.py (its $ lines are run by the tests).
More: Deleting an account and Currencies and exchange rates.