ByteDance-Seed / ByteDance-Seed/DAComp
Incomplete Rubric for a few examples
- Dominant language
- Python
- Stars
- 441
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
The DE Arch Evaluation rubric is incomplete for following example ids:
1. [dacomp-de-arch-006](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L6).
2. [dacomp-de-arch-007](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L7).
3. [dacomp-de-arch-017](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L17).
4. [dacomp-de-arch-012](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L12).
5. [dacomp-de-arch-002](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L2).
6. [dacomp-de-arch-016](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L16).
Please update the file if possible or guide me to the right location for rubric information.
Regards,
Harsha
Contributor guide
No contributing guide indexed for this repository
Research direction
Open dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl and inspect entries 002, 006, 007, 012, 016, and 017 linked in the issue. Compare them with nearby rubric entries and identify which rubric information is incomplete. Done means all six examples contain complete rubric data and the evaluation suite still accepts the updated file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100