JetBrains-Research / JetBrains-Research/EnvBench

How to understand results.jsonl

Open
#7 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I ma using this to see per file pass rate across runs but i am unable to come across the 6% pass rat whatsoever

jq -s '
def miss_cnt(r):
((r.pyright.generalDiagnostics // {})
| to_entries
| map(.value.rule // "" | tostring | ascii_downcase)
| map(select(.=="reportmissingimports" or .=="reportmissingmodulesource"))
| length);

{ total: length,
success: map(select(miss_cnt(.) == 0)) | length } as $m
| $m + { rate: ($m.success / $m.total) }
' results.jsonl

Results :-

bash_agent-4o.jsonl : {
"total": 327,
"success": 39,
"rate": 0.11926605504587157
}
bash_agent-4o-mini.jsonl : {
"total": 327,
"success": 38,
"rate": 0.1162079510703364
}
installamatic_agent-4o.jsonl : {
"total": 329,
"success": 178,
"rate": 0.541033434650456
}
installamatic_agent-4o-mini.jsonl : {
"total": 329,
"success": 201,
"rate": 0.6109422492401215
}
zero_shot-4o.jsonl : {
"total": 329,
"success": 191,
"rate": 0.5805471124620061
}
zero_shot-4o-mini.jsonl : {
"total": 329,
"success": 161,
"rate": 0.48936170212765956
}

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied jq expression and the results.jsonl outputs, checking how miss_cnt selects reportmissingimports and reportmissingmodulesource diagnostics and how success and rate are derived. Done means documenting why these results do not show the expected 6% pass rate, including which input or interpretation accounts for the difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
data
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.