All nightly moonlight tests failing with sentencepiece error
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
All nightly moonlight tests failing with error despite sentenepiece being in direct rl dependencies:
```
ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a `tokenizers` library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.
```
**Steps/Code to reproduce bug**
Tests failing off main branch:
llm_grpo_moonlight_16ba3b_4n8g_megatron_fp8_e2e
llm_grpo_moonlight_16ba3b_4n8g_megatron
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple etc.
**Additional context**
Failures after PR https://github.com/NVIDIA-NeMo/RL/pull/2489 merged
Contributor guide
Assessment
This issue has not been assessed yet.