JetBrains-Research / JetBrains-Research/EnvBench
How to understand results.jsonl
- Dominant language
- Python
- Stars
- 38
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
I ma using this to see per file pass rate across runs but i am unable to come across the 6% pass rat whatsoever
jq -s '
def miss_cnt(r):
((r.pyright.generalDiagnostics // {})
| to_entries
| map(.value.rule // "" | tostring | ascii_downcase)
| map(select(.=="reportmissingimports" or .=="reportmissingmodulesource"))
| length);
{ total: length,
success: map(select(miss_cnt(.) == 0)) | length } as $m
| $m + { rate: ($m.success / $m.total) }
' results.jsonl
Results :-
bash_agent-4o.jsonl : {
"total": 327,
"success": 39,
"rate": 0.11926605504587157
}
bash_agent-4o-mini.jsonl : {
"total": 327,
"success": 38,
"rate": 0.1162079510703364
}
installamatic_agent-4o.jsonl : {
"total": 329,
"success": 178,
"rate": 0.541033434650456
}
installamatic_agent-4o-mini.jsonl : {
"total": 329,
"success": 201,
"rate": 0.6109422492401215
}
zero_shot-4o.jsonl : {
"total": 329,
"success": 191,
"rate": 0.5805471124620061
}
zero_shot-4o-mini.jsonl : {
"total": 329,
"success": 161,
"rate": 0.48936170212765956
}
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the supplied jq expression and the results.jsonl outputs, checking how miss_cnt selects reportmissingimports and reportmissingmodulesource diagnostics and how success and rate are derived. Done means documenting why these results do not show the expected 6% pass rate, including which input or interpretation accounts for the difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100