apache / apache/lucene

PR benchmark bot for Lucene?

Open
#16,398 1 comment 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

While working on performance optimization features in lucene, I see we usually end up writing our own JMH micro-benchmark(we have a lot of them now) or run existing luceneutil benchmarks locally. And regressions are often only caught later during nightlies.

I was wondering if we could provide a way for users to trigger lightweight benchmarks directly from a PR? Someone could write a command like below, and benchmarks would automatically run in the background and post results on the PR comparing against the baseline:

```
/benchmark jmh # Run an existing JMH benchmark
/benchmark jmh "Write a benchmark to test my PR changes" # passing a prompt to AI model,
# it writes a temporary benchmark that doesn't need to checked in

/benchmark search # luceneutil search-only (on top of existing data)
/benchmark full # luceneutil index + search (default 1M/10M docs?)
/benchmark full --data wikimedium5m # explicit corpus size
```

This would make it easy for anyone(author or reviewer) to run benchmarks and catch regressions before merge.

I know this requires a dedicated machine to run the benchmarks, so I'm not sure how that part would work. Maybe something similar to what we already have for the luceneutil nightlies or a self-hosted GitHub Actions runner?

I'm not sure if folks have already thought about this, but I'd love to hear opinions.

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified. Start by reviewing the existing luceneutil nightlies, JMH benchmark setup, and possible GitHub Actions or self-hosted runner arrangements. Done would require an agreed scope for the benchmark command, baseline comparison, resource handling, and PR reporting before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, java
Domain
ci-cd, developer-experience, performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.