JetBrains-Research / JetBrains-Research/EnvBench
JVM evaluation script for multimodule projects
- Dominant language
- Python
- Stars
- 38
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
I express my sincere respect for gathering such a large amount of data and for preparing a benchmark framework that allows for flexible configuration changes.
You are measuring pass@1 by executing `mvnw compile` with the provided jvm_build.sh, but this does not seem to be suitable for some multimodule projects like google/guava. Have you tried `./mvnw install`?
Official example: https://books.sonatype.com/mvnex-book/reference/multimodule-sect-building-multimodule.html
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with jvm_build.sh and inspect how it invokes ./mvnw compile for projects such as google/guava. Compare that behavior with the linked multimodule Maven guidance and determine whether the evaluation should use ./mvnw install; done means multimodule projects are evaluated successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, shell
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100