aws-samples / aws-samples/sagemaker-trainium-examples
Update transformers version for BERT pretraining
Open
- Dominant language
- Python
- Stars
- 18
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
The example to pretrain BERT on Trainium is based on the original example from the AWS Neuron SDK [here](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/bert.html#bert-large-compilation-limitations ).
This calls out a known issues with some versions of Transformers, especially with the BERT model downloading from the Hugging Face Hub.
I'm using transformers version 4.20.0, and now the compiler working.
Contributor guide
Assessment
This issue has not been assessed yet.