Remove kernels-community/mamba-ssm autocast warning filter once the Hub kernel build stops calling `torch.get_autocast_gpu_dtype()`
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### What
`tests_dev` (CI job installing `accelerate`/`transformers`/`datasets`/`peft` from git main) emits repeated `DeprecationWarning`s from the NemotronH (Nemotron 3) parametrized case:
```
tests/test_dpo_trainer.py: 18 warnings
tests/test_sft_trainer.py: 18 warnings
/github/home/.cache/huggingface/hub/kernels--kernels-community--mamba-ssm/snapshots/.../build/torch213-cxx11-cu130-x86_64-linux/ops/triton/ssd_combined.py:1511: DeprecationWarning: torch.get_autocast_gpu_dtype() is deprecated. Please use torch.get_autocast_dtype('cuda') instead. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/init.cpp:1007.)
dtype = torch.get_autocast_gpu_dtype()
```
Affected tests: `tests/test_dpo_trainer.py::TestDPOTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM-nano]` and `tests/test_sft_trainer.py::TestSFTTrainer::test_train[trl-internal-testing/tiny-NemotronHForCausalLM-nano]` (see run https://github.com/huggingface/trl/actions/runs/30240017065/job/89895088560).
### Root cause
The warning originates in the Hub-hosted kernel `kernels-community/mamba-ssm` — an auto-fetched build of `state-spaces/mamba`'s `mamba_ssm/ops/triton/ssd_combined.py`, used by transformers' NemotronH Mamba2 mixer fast path (`transformers.models.nemotron_h.modeling_nemotron_h`, via `lazy_load_kernel("mamba-ssm")`). That kernel still calls the deprecated `torch.get_autocast_gpu_dtype()` instead of `torch.get_autocast_dtype('cuda')`. This is not TRL code, not transformers code, not even the `kernels` integration glue — it's inside the kernel build pulled from the HF Hub.
It only shows up in `tests_dev`, not in the regular `tests (3.12)` job, even though both jobs resolve identical `torch` (2.13.0) and `kernels` (0.15.2) versions and both select the same fast (kernelized) Mamba2 path in this run (neither log shows a "falling back to the naive implementation" message). The difference is that only `tests_dev`'s run has the deprecated line actually reached, which is gated by `if torch.is_autocast_enabled():` inside the kernel — i.e. only `tests_dev` runs this forward/backward pass under CUDA autocast. Since `tests_dev` is the only job installing `accelerate`/`transformers`/`datasets`/`peft` from git main, one of those unreleased dependencies most likely changed when/whether autocast gets engaged for this training path. Not yet bisected to the exact commit/library — tracked as a follow-up in this issue.
### Upstream status
Not yet reported. No existing open issue found on `state-spaces/mamba` or in the `kernels-community/mamba-ssm` Hub repo discussions.
### Workaround in place
Suppressed via `filterwarnings` in `pyproject.toml`:
```toml
"ignore:torch.get_autocast_gpu_dtype\\(\\) is deprecated:DeprecationWarning",
```
### Removal criterion
The emitter is a Hub-hosted kernel build (`kernels-community/mamba-ssm`), resolved at runtime and not version-constrained anywhere in TRL's metadata. There is no pinned older version left to support, so (unlike our pip-dependency filters) the criterion is not a floor comparison: the filter can go as soon as the upstream kernel is fixed and the fixed build is what CI resolves.
Note that the transformers regression is *not* the right trigger. #47452 only changed **whether** the kernel path is reached; restoring the `use_mamba_kernels` guard would silence `tests_dev` today, but TRL supports `transformers >= 4.56.2` with no upper bound, so any released transformers shipping without the guard would keep reaching the kernel. Tie removal to the kernel fix; track the transformers guard as its own upstream item.
### Action items
- [ ] Report upstream (`state-spaces/mamba` and/or a discussion on the `kernels-community/mamba-ssm` Hub repo).
- [x] Bisect which git-main dependency (accelerate/transformers/peft/datasets) changed the autocast behavior for the NemotronH Mamba2 path, to know whether this will also start appearing in the regular `tests` job once that dependency's next release ships.
- [ ] Remove the `filterwarnings` entry once the fixed kernel build is what CI resolves, across the torch build variants in use (e.g. `torch213-cxx11-cu130-x86_64-linux`), confirmed by a warning-free `tests_dev` run.
Contributor guide
Research direction
Start in pyproject.toml by reviewing the filterwarnings entry and the affected NemotronH cases in tests/test_dpo_trainer.py and tests/test_sft_trainer.py. Run the tests_dev job across the torch build variants after the kernels-community/mamba-ssm Hub build is fixed. Done means the fixed kernel is resolved, tests_dev is warning-free, and the filter can be removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100