HarperFast / HarperFast/harper-pro

Trigger the self-hosted harper-bench nightlies from the bench host instead of GitHub cron

Open
#773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Problem

GitHub's scheduled-workflow delay is unbounded, and the `harper-bench` nightlies have no protection against it. On 2026-08-27 every scheduled workflow in this repo fired roughly 3 hours late:

| Workflow | Cron (UTC) | Fired |
| --- | --- | --- |
| Smoke Tests | `0 6` | 09:09 |
| Large-Data Stress Tests | `0 6` | 09:03 |
| Integration Tests | `0 7` | 10:11 |
| Sync Core | `0 7` | 10:23 |
| YCSB Cluster Nightly | `0 10` | 13:34 |
| Companion Check | `*/15` | fired 5 times in 24h instead of 96 |

The typical delay is 5–35 minutes, so the nightlies normally land where the cron intends. But a slip of this size defeats any start time: the YCSB cluster benchmark ran until 13:57 UTC — 07:57 in the bench host's local timezone, well into the working day, on a box that is also somebody's desktop.

Moving the crons earlier (HarperFast/harper-pro#774, HarperFast/harper#2362) buys headroom for the ordinary delay. It cannot fix this.

## Proposal

Drop `schedule:` from the four `harper-bench` workflows and let the bench host trigger them:

- harper-pro `stress-large-data.yaml`, `ycsb-cluster-nightly.yml`
- harper `perf-benchmarks-nightly.yml`, `large-deploy-test.yml`

Keep `workflow_dispatch` (the workflows already have it, and it stays the manual path). Add a `systemd --user` timer on the bench host that calls `gh workflow run` for each workflow at its slot.

The host is the natural place for this: `bench-runner-supervisor.sh` already runs there as a `systemd --user` unit and already polls both repos for queued `harper-bench` jobs, so the credentials, the unit pattern, and the repo list all exist. The timer is a sibling unit, not new infrastructure.

Two properties fall out of it:

1. **The trigger is on time.** A local timer fires when it says it will; the job then queues for the supervisor, which is the only wait that reflects real contention.
2. **The times become DST-aware.** GitHub crons are UTC-only, so every schedule shifts an hour against local time twice a year — the current set is tuned for MDT and lands an hour earlier in MST. A `systemd` `OnCalendar=` in the host's timezone does not drift.

## Out of scope

- **Supervisor downtime.** On 2026-08-09 a queued YCSB run sat for 24 hours and was auto-cancelled; the next night's stress run waited 3h45 for the runner. That is a liveness gap in the supervisor, not a scheduling one, and wants its own issue (health check / alert on a job queued past a threshold).
- **Job timeouts.** The four `timeout-minutes` values sum to 8 hours against measured runs of 10–40 minutes. Tightening them bounds a wedged job, independent of what triggers it.

## Alternatives considered

- **Earlier crons only** — what HarperFast/harper-pro#774 and HarperFast/harper#2362 do. Necessary, not sufficient: it moves the window but keeps the unbounded delay.
- **A repository_dispatch webhook from an external scheduler** — same on-time property, but adds a service to own. The bench host already has a supervisor process and has to be up for these jobs to run at all, so it is the cheaper home.

Contributor guide

Open the contributing guide

Research direction

Start with harper-pro stress-large-data.yaml and ycsb-cluster-nightly.yml, then inspect the referenced harper workflows perf-benchmarks-nightly.yml and large-deploy-test.yml. Read bench-runner-supervisor.sh and its systemd --user unit to follow the existing host pattern. Done means the four workflows retain workflow_dispatch, no longer use schedule, and sibling timers trigger each workflow at its local-time slot.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, linux, shell
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.