apache / apache/datafusion

Simplify CI benchmark comparison development

Open
#9,638 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The present CI benches use the `issue_comment` event type, and are triggered by a `/benchmark` keyword in a PR comment.

This now works, but there are some issues with the dev experience around it, as demonstrated in https://github.com/apache/arrow-datafusion/issues/9620.

In particular the `issue_comment` event type will use the workflow as it is on the default (i.e. main) branch, meaning any changes introduced to it via a PR can't be tested until the PR is merged, which is suboptimal.

### Describe the solution you'd like

The proper way to test new changes to this workflow is to use the `pull_request` event type, which would enable a tight feedback loop during development (i.e. testing the workflow version that is in the PR code). However to my knowledge this event type can't be triggered by a PR comment then (the sole purpose of this is to reduce noise, and don't benchmark/comment on every PR/change).

Instead, the `pull_request`-based workflow can be made [conditional upon a label](https://stackoverflow.com/a/74829754).

In addition, the `Benchmarks` workflow could be triggered through a workflow_call event from inside the `Rust` workflow (which itself has a `pull_request` event trigger). This would ensure that the benches could be run only if the build/tests pass. One downside to this though is that the whole chain would need to be kickstarted again (by a push or manually) even once the label has been added to get the first benchmark results. Every subsequent run would then perform benchmarks (as long as the label is present).

### Describe alternatives you've considered

Continue with multi-step development, since this is probably not going to be changed that often.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the existing CI benchmark workflow and the Rust workflow, focusing on their current triggers and how benchmark results are posted. Compare the proposed pull_request, label, and workflow_call paths; done means benchmark runs are gated as described and workflow changes can be tested from the pull request.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.