ByteDance-Seed / ByteDance-Seed/DAComp

Incomplete Rubric for a few examples

Open Beginner friendly
#9 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
441
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Hello,

The DE Arch Evaluation rubric is incomplete for following example ids:

1. [dacomp-de-arch-006](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L6).
2. [dacomp-de-arch-007](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L7).
3. [dacomp-de-arch-017](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L17).
4. [dacomp-de-arch-012](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L12).
5. [dacomp-de-arch-002](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L2).
6. [dacomp-de-arch-016](https://github.com/ByteDance-Seed/DAComp/blob/027ccaf20f0d3743d9291fd575d9bc785a5fe3db/dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl#L16).

Please update the file if possible or guide me to the right location for rubric information.

Regards,
Harsha

Contributor guide

No contributing guide indexed for this repository

Research direction

Open dacomp-de/evaluation_suite_arch/gold/dacomp-arch-gold.jsonl and inspect entries 002, 006, 007, 012, 016, and 017 linked in the issue. Compare them with nearby rubric entries and identify which rubric information is incomplete. Done means all six examples contain complete rubric data and the evaluation suite still accepts the updated file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.