NVIDIA-Merlin / NVIDIA-Merlin/Transformers4Rec
[BUG] Trainer have no attribute 'use_cuda_amp'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 165
- Avg merge
- 1m
- Merged PRs (30d)
- 2
Description
Bug description
Trainer.train() when finishing training return:
AttributeError: 'Trainer' object has no attribute 'use_cuda_amp'
During handling of the above exception, another exception occurred:
AttributeError: 'Trainer' object has no attribute 'use_amp'
Steps/Code to reproduce bug
Train a model with these configurations:
training_args = T4RecTrainingArguments(
output_dir=lb_out.as_posix(),
max_sequence_length=20,
data_loader_engine='nvtabular',
num_train_epochs=1,
dataloader_drop_last=False,
per_device_train_batch_size=256,
per_device_eval_batch_size=16,
gradient_accumulation_steps=1,
learning_rate=0.0005,
report_to=[],
save_strategy='no',
logging_steps=500
)
trainer = Trainer(
model=model,
args=training_args,
schema=schema,
compute_metrics=True,
)
Expected behavior
Model finish training without returning error
Environment details
- Transformers4Rec version: 23.08.00
- Platform:Ubuntu 22.04.5 LTS
- Python version:3.10.16
- Huggingface Transformers version:4.45.0
- PyTorch version (GPU?):2.4.1
- Tensorflow version (GPU?):
Additional context
It seems that transformers have remove CUDA amp entirely for CPU amp. huggingface/transformers#27760
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Trainer.train() and inspect how the Trainer handles AMP attributes when training finishes, using the provided T4RecTrainingArguments configuration to reproduce the failure. Check the related PyTorch and Hugging Face Transformers behavior, and consider the issue done when training completes without either AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100