haskell / haskell/haskell-language-server

Improving the Benchmark Suite

Open
#5,006 1 comment 2 reactions 0 assignees View on GitHub
status: in discussion
Dominant language
Haskell
Stars
3k
Forks
455
Avg merge
2d 19h
Merged PRs (30d)
11

Description

The benchmark suite is a highly flexible performance benchmarking framework, centered
around Shake build targets. However, it has high cognitive load for HLS non-experts. In
particular, it:

* Has multiple interconnected projects: `bench`, `ghcide-bench` and `shake-bench`.
* Constructs an entire Shake build system
* Outputs large hierarchical CSV
* Has high levels of abstraction

That said, I propose the following changes:

* Consolidate `bench`, `ghcide-bench` and `shake-bench` into a single `bench` project
* Completely remove `shake`
* Replace orchestration/statistical processing with `tasty-bench`, if reasonably fits
* Use familiar benchmarking terminology:
* Examples -> Projects or Harnesses
* Experiments -> Benchmarks
* Configurations -> separate into two distinct concepts:
* Test run configuration
* HLS plugin set

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by mapping the three named projects—bench, ghcide-bench, and shake-bench—and reviewing how they construct Shake builds and produce hierarchical CSV output. Compare the current orchestration with tasty-bench, then define completion as one consolidated bench project with the proposed terminology and no Shake dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, performance, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.