JetBrains-Research / JetBrains-Research/EnvBench

JVM evaluation script for multimodule projects

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I express my sincere respect for gathering such a large amount of data and for preparing a benchmark framework that allows for flexible configuration changes.

You are measuring pass@1 by executing `mvnw compile` with the provided jvm_build.sh, but this does not seem to be suitable for some multimodule projects like google/guava. Have you tried `./mvnw install`?

Official example: https://books.sonatype.com/mvnex-book/reference/multimodule-sect-building-multimodule.html

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with jvm_build.sh and inspect how it invokes ./mvnw compile for projects such as google/guava. Compare that behavior with the linked multimodule Maven guidance and determine whether the evaluation should use ./mvnw install; done means multimodule projects are evaluated successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
build-system
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.