AnswerDotAI / AnswerDotAI/ModernBERT
`IndexError: tuple index out of range` in `input_ids` during pre-training
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Following the [instructions](https://github.com/AnswerDotAI/ModernBERT?tab=readme-ov-file#setup) in the README, I am working on pre-training from scratch. I ran training with the composer framework using the `yamls/main/flex-bert-base.yaml` config and with the c4 dataset `./my-copy-c4`.
[note that I verified that the dataloader works fine following the [instructions](https://github.com/mosaicml/examples/tree/main/examples/benchmarks/bert#test-the-dataloader).]
However, I encountered the following error during training:
```
IndexError: tuple index out of range
File "/.../ModernBERT/src/bert_layers/embeddings.py", line 153, in forward
position_ids = self.position_ids[:, 0 : input_ids.shape[1]]
~~~~~~~~~~~~~~~^^^
```
## Steps to Reproduce
1. Prepare the c4 dataset.
2. Set up the conda environment per instructions.
3. Run training with `composer main.py yamls/main/flex-bert-base.yaml`
4. The error occurs during training in `bert_layers/embeddings.py`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.