ByteDance-Seed / ByteDance-Seed/Bagel
GenEval setup
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for your great work!
**GenEval** benchmark for proper operation requires old version of packages.
Trying to install the specified in `EVAL.md` on top of environment with `torch>=2.5` leads to errors somewhere in the CUDA backend when trying to execute this [line](https://github.com/ByteDance-Seed/Bagel/blob/7026cfa0a4df274460d0b0b990117398a4ec6fca/eval/gen/geneval/evaluation/evaluate_images_mp.py#L143):
```shell
frame #11: THPVariable_subclass_dealloc(_object*) + 0x300 (0x7f129f58f090 in /home/dkuznedelev/uv/llm/lib/python3.12/site-packa
ges/torch/lib/libtorch_python.so)
frame #51: + 0x29d90 (0x7f135fae2d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #52: __libc_start_main + 0x80 (0x7f135fae2e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #53: _start + 0x29 (0x6000a9 in /home/dkuznedelev/uv/llm/bin/python3)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```
The resolution is to use a separate environment following the steps provided in this [issue](https://github.com/djghosh13/geneval/issues/12).
I think it would be useful to add this information to `EVAL.md`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with EVAL.md and the GenEval entry point at eval/gen/geneval/evaluation/evaluate_images_mp.py:143. Review the separate-environment steps in the linked geneval issue, then document the required setup and confirm that the instructions address the torch>=2.5 CUDA failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100