huggingface / huggingface/trl

Remove kernels-community/mamba-ssm autocast warning filter once the Hub kernel build stops calling `torch.get_autocast_gpu_dtype()`

Open
#6,555 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### What

`tests_dev` (CI job installing `accelerate`/`transformers`/`datasets`/`peft` from git main) emits repeated `DeprecationWarning`s from the NemotronH (Nemotron 3) parametrized case:

```
tests/test_dpo_trainer.py: 18 warnings
tests/test_sft_trainer.py: 18 warnings
/github/home/.cache/huggingface/hub/kernels--kernels-community--mamba-ssm/snapshots/.../build/torch213-cxx11-cu130-x86_64-linux/ops/triton/ssd_combined.py:1511: DeprecationWarning: torch.get_autocast_gpu_dtype() is deprecated. Please use torch.get_autocast_dtype('cuda') instead. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/init.cpp:1007.)
dtype = torch.get_autocast_gpu_dtype()
```

Affected tests: `tests/test_dpo_trainer.py::TestDPOTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM-nano]` and `tests/test_sft_trainer.py::TestSFTTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM-nano]` (see run https://github.com/huggingface/trl/actions/runs/30240017065/job/89895088560).

### Root cause

The warning originates in the Hub-hosted kernel `kernels-community/mamba-ssm` — an auto-fetched build of `state-spaces/mamba`'s `mamba_ssm/ops/triton/ssd_combined.py`, used by transformers' NemotronH Mamba2 mixer fast path (`transformers.models.nemotron_h.modeling_nemotron_h`, via `lazy_load_kernel("mamba-ssm")`). That kernel still calls the deprecated `torch.get_autocast_gpu_dtype()` instead of `torch.get_autocast_dtype('cuda')`. This is not TRL code, not transformers code, not even the `kernels` integration glue — it's inside the kernel build pulled from the HF Hub.

It only shows up in `tests_dev`, not in the regular `tests (3.12)` job, even though both jobs resolve identical `torch` (2.13.0) and `kernels` (0.15.2) versions and both select the same fast (kernelized) Mamba2 path in this run (neither log shows a "falling back to the naive implementation" message). The difference is that only `tests_dev`'s run has the deprecated line actually reached, which is gated by `if torch.is_autocast_enabled():` inside the kernel — i.e. only `tests_dev` runs this forward/backward pass under CUDA autocast. Since `tests_dev` is the only job installing `accelerate`/`transformers`/`datasets`/`peft` from git main, one of those unreleased dependencies most likely changed when/whether autocast gets engaged for this training path. Not yet bisected to the exact commit/library — tracked as a follow-up in this issue.

### Upstream status

Not yet reported. No existing open issue found on `state-spaces/mamba` or in the `kernels-community/mamba-ssm` Hub repo discussions.

### Workaround in place

Suppressed via `filterwarnings` in `pyproject.toml`:

```toml
"ignore:torch.get_autocast_gpu_dtype\\(\\) is deprecated:DeprecationWarning",
```

### Removal criterion

The emitter is a Hub-hosted kernel build (`kernels-community/mamba-ssm`), resolved at runtime and not version-constrained anywhere in TRL's metadata. There is no pinned older version left to support, so (unlike our pip-dependency filters) the criterion is not a floor comparison: the filter can go as soon as the upstream kernel is fixed and the fixed build is what CI resolves.

Note that the transformers regression is *not* the right trigger. #47452 only changed **whether** the kernel path is reached; restoring the `use_mamba_kernels` guard would silence `tests_dev` today, but TRL supports `transformers >= 4.56.2` with no upper bound, so any released transformers shipping without the guard would keep reaching the kernel. Tie removal to the kernel fix; track the transformers guard as its own upstream item.

### Action items

- [ ] Report upstream (`state-spaces/mamba` and/or a discussion on the `kernels-community/mamba-ssm` Hub repo).
- [x] Bisect which git-main dependency (accelerate/transformers/peft/datasets) changed the autocast behavior for the NemotronH Mamba2 path, to know whether this will also start appearing in the regular `tests` job once that dependency's next release ships.
- [ ] Remove the `filterwarnings` entry once the fixed kernel build is what CI resolves, across the torch build variants in use (e.g. `torch213-cxx11-cu130-x86_64-linux`), confirmed by a warning-free `tests_dev` run.

Contributor guide

Open the contributing guide

Research direction

Start in pyproject.toml by reviewing the filterwarnings entry and the affected NemotronH cases in tests/test_dpo_trainer.py and tests/test_sft_trainer.py. Run the tests_dev job across the torch build variants after the kernels-community/mamba-ssm Hub build is fixed. Done means the fixed kernel is resolved, tests_dev is warning-free, and the filter can be removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, pytorch
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.