ByteDance-Seed / ByteDance-Seed/Bagel

GenEval setup

Open
#242 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Thank you for your great work!

**GenEval** benchmark for proper operation requires old version of packages.

Trying to install the specified in `EVAL.md` on top of environment with `torch>=2.5` leads to errors somewhere in the CUDA backend when trying to execute this [line](https://github.com/ByteDance-Seed/Bagel/blob/7026cfa0a4df274460d0b0b990117398a4ec6fca/eval/gen/geneval/evaluation/evaluate_images_mp.py#L143):
```shell
frame #11: THPVariable_subclass_dealloc(_object*) + 0x300 (0x7f129f58f090 in /home/dkuznedelev/uv/llm/lib/python3.12/site-packa
ges/torch/lib/libtorch_python.so)

frame #51: + 0x29d90 (0x7f135fae2d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #52: __libc_start_main + 0x80 (0x7f135fae2e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #53: _start + 0x29 (0x6000a9 in /home/dkuznedelev/uv/llm/bin/python3)

terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```
The resolution is to use a separate environment following the steps provided in this [issue](https://github.com/djghosh13/geneval/issues/12).

I think it would be useful to add this information to `EVAL.md`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with EVAL.md and the GenEval entry point at eval/gen/geneval/evaluation/evaluate_images_mp.py:143. Review the separate-environment steps in the linked geneval issue, then document the required setup and confirm that the instructions address the torch>=2.5 CUDA failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.