NVIDIA / NVIDIA/Megatron-LM

request GDN to support inference

Open
#3,881 5 comments 2 reactions 2 assignees Claimed by @YangFei1990 View on GitHub
community-request enhancement waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Tagging @mcore-oncall for visibility.

**Is your feature request related to a problem? Please describe.**
We are using Megatron-Core's experimental `gated_delta_net` attention variant in a GPT decoder model for training, but inference is currently blocked by an explicit `NotImplementedError` in [`megatron/core/ssm/gated_delta_net.py`](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/ssm/gated_delta_net.py).

The current implementation rejects inference unconditionally:

```python
if inference_context is not None:
assert (
inference_context.is_static_batching()
), "GDN does not currently support dynamic inference batching."
assert not self.config.sequence_parallel
# TODO: support inference
raise NotImplementedError("GDN does not support inference for now.")

if packed_seq_params is not None:
# TODO: support packed sequence
raise NotImplementedError("GDN does not support packed sequence for now.")

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.