HarperFast / HarperFast/harper-pro
Trigger the self-hosted harper-bench nightlies from the bench host instead of GitHub cron
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Problem
GitHub's scheduled-workflow delay is unbounded, and the `harper-bench` nightlies have no protection against it. On 2026-08-27 every scheduled workflow in this repo fired roughly 3 hours late:
| Workflow | Cron (UTC) | Fired |
| --- | --- | --- |
| Smoke Tests | `0 6` | 09:09 |
| Large-Data Stress Tests | `0 6` | 09:03 |
| Integration Tests | `0 7` | 10:11 |
| Sync Core | `0 7` | 10:23 |
| YCSB Cluster Nightly | `0 10` | 13:34 |
| Companion Check | `*/15` | fired 5 times in 24h instead of 96 |
The typical delay is 5–35 minutes, so the nightlies normally land where the cron intends. But a slip of this size defeats any start time: the YCSB cluster benchmark ran until 13:57 UTC — 07:57 in the bench host's local timezone, well into the working day, on a box that is also somebody's desktop.
Moving the crons earlier (HarperFast/harper-pro#774, HarperFast/harper#2362) buys headroom for the ordinary delay. It cannot fix this.
## Proposal
Drop `schedule:` from the four `harper-bench` workflows and let the bench host trigger them:
- harper-pro `stress-large-data.yaml`, `ycsb-cluster-nightly.yml`
- harper `perf-benchmarks-nightly.yml`, `large-deploy-test.yml`
Keep `workflow_dispatch` (the workflows already have it, and it stays the manual path). Add a `systemd --user` timer on the bench host that calls `gh workflow run` for each workflow at its slot.
The host is the natural place for this: `bench-runner-supervisor.sh` already runs there as a `systemd --user` unit and already polls both repos for queued `harper-bench` jobs, so the credentials, the unit pattern, and the repo list all exist. The timer is a sibling unit, not new infrastructure.
Two properties fall out of it:
1. **The trigger is on time.** A local timer fires when it says it will; the job then queues for the supervisor, which is the only wait that reflects real contention.
2. **The times become DST-aware.** GitHub crons are UTC-only, so every schedule shifts an hour against local time twice a year — the current set is tuned for MDT and lands an hour earlier in MST. A `systemd` `OnCalendar=` in the host's timezone does not drift.
## Out of scope
- **Supervisor downtime.** On 2026-08-09 a queued YCSB run sat for 24 hours and was auto-cancelled; the next night's stress run waited 3h45 for the runner. That is a liveness gap in the supervisor, not a scheduling one, and wants its own issue (health check / alert on a job queued past a threshold).
- **Job timeouts.** The four `timeout-minutes` values sum to 8 hours against measured runs of 10–40 minutes. Tightening them bounds a wedged job, independent of what triggers it.
## Alternatives considered
- **Earlier crons only** — what HarperFast/harper-pro#774 and HarperFast/harper#2362 do. Necessary, not sufficient: it moves the window but keeps the unbounded delay.
- **A repository_dispatch webhook from an external scheduler** — same on-time property, but adds a service to own. The bench host already has a supervisor process and has to be up for these jobs to run at all, so it is the cheaper home.
Contributor guide
Research direction
Start with harper-pro stress-large-data.yaml and ycsb-cluster-nightly.yml, then inspect the referenced harper workflows perf-benchmarks-nightly.yml and large-deploy-test.yml. Read bench-runner-supervisor.sh and its systemd --user unit to follow the existing host pattern. Done means the four workflows retain workflow_dispatch, no longer use schedule, and sibling timers trigger each workflow at its local-time slot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, linux, shell
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100