AnswerDotAI / AnswerDotAI/ModernBERT

`IndexError: tuple index out of range` in `input_ids` during pre-training

Open
#199 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Following the [instructions](https://github.com/AnswerDotAI/ModernBERT?tab=readme-ov-file#setup) in the README, I am working on pre-training from scratch. I ran training with the composer framework using the `yamls/main/flex-bert-base.yaml` config and with the c4 dataset `./my-copy-c4`.
[note that I verified that the dataloader works fine following the [instructions](https://github.com/mosaicml/examples/tree/main/examples/benchmarks/bert#test-the-dataloader).]

However, I encountered the following error during training:

```
IndexError: tuple index out of range

File "/.../ModernBERT/src/bert_layers/embeddings.py", line 153, in forward
position_ids = self.position_ids[:, 0 : input_ids.shape[1]]
~~~~~~~~~~~~~~~^^^
```

## Steps to Reproduce

1. Prepare the c4 dataset.
2. Set up the conda environment per instructions.
3. Run training with `composer main.py yamls/main/flex-bert-base.yaml`
4. The error occurs during training in `bert_layers/embeddings.py`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.