NVIDIA-Merlin / NVIDIA-Merlin/Transformers4Rec

[BUG] `load_model_trainer_states_from_checkpoint()` function is broken

Open Beginner friendly
#806 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug status/needs-triage
Dominant language
Python
Stars
1.3k
Forks
165
Avg merge
1m
Merged PRs (30d)
2

Description

Bug description

The transformers4rec.torch.trainer.load_model_trainer_states_from_checkpoint() function can only load t4rec_model_class.pkl, all the other loads are broken. The utils.serialization.load() function is called 4 times in the function, but only one of them successfully loads, the other throw errors. I believe this is because load() is expecting an _io.BufferedReader object (resulting from open()) instead of a path.

Correctly implemented:
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/5d59d14562cff667ce624f4a2a619a9803b7c4af/transformers4rec/torch/trainer.py#L744-L746

Incorrectly implemented:
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/5d59d14562cff667ce624f4a2a619a9803b7c4af/transformers4rec/torch/trainer.py#L753
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/5d59d14562cff667ce624f4a2a619a9803b7c4af/transformers4rec/torch/trainer.py#L757
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/5d59d14562cff667ce624f4a2a619a9803b7c4af/transformers4rec/torch/trainer.py#L766

Steps/Code to reproduce bug

Run the following code with a Model checkpoint at /checkpoint:

from transformers4rec import torch as tr

trainer = tr.trainer.Trainer(model=tr.model.base.Model(), args=None)
trainer.load_model_trainer_states_from_checkpoint('/checkpoint')
Expected behavior

The function call shouldn't be erroring.

Environment details
  • Transformers4Rec version: 23.12.0
  • Platform: Linux
  • Python version: 3.10.12
  • Huggingface Transformers version: 4.27.1
  • PyTorch version (GPU?): 2.1.2 (no)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in transformers4rec/torch/trainer.py at load_model_trainer_states_from_checkpoint(), especially the working load call around lines 744-746 and the other calls around lines 753, 757, and 766. Reproduce with the provided checkpoint-loading example, then verify that all trainer states load without errors rather than only t4rec_model_class.pkl.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.