NVIDIA / NVIDIA/Megatron-LM

tools/run_vlm_text_generation.py cannot be imported: it uses megatron.inference.text_generation, removed by the legacy-inference deprecation

Open
#6,860 3 comments 0 reactions 1 assignee Assigned to @RPrenger View on GitHub
community-request waiting-on-maintainers
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

`tools/run_vlm_text_generation.py` cannot be imported. Lines 19-20 are

```python
from megatron.inference.text_generation.api import generate_and_post_process
from megatron.inference.text_generation.forward_step import ForwardStep
```

but `megatron/inference/` now contains only `__init__.py` and `utils.py` — the `text_generation`
subpackage was removed by `4d5135359` ("ADLR/megatron-lm!3818 - Deprecate legacy inference"). Both imports
are at module top level, so the script raises `ModuleNotFoundError` before it parses an argument.

It is also broken in a second, independent way that predates that deletion, which is the part that suggests
nobody has run it in a long time: `VLMForwardStep.__init__` calls
`super().__init__(model, max_batch_size, max_sequence_length)` (line 170), while at the last commit where
the import resolved the base constructor was `def __init__(self, model, inference_context)`. So restoring
the import alone would not make the file work.

Tagging @NVIDIA/mcore-oncall.

**Steps/Code to reproduce bug**

No GPU, no data, no checkpoint needed:

```bash
python -c "import tools.run_vlm_text_generation"
# ModuleNotFoundError: No module named 'megatron.inference.text_generation'

git ls-tree HEAD megatron/inference/
# only __init__.py and utils.py
```

**Expected behavior**

Either the tool runs, or it is not shipped. Right now it is shipped and cannot be imported.

**Additional context**

I am not sending a PR because the right fix is a decision I cannot make for you. Three options, and I am
happy to do any of them:

1. **Port it** to `megatron.core.inference` — the largest option, and it needs someone who knows what the
tool is meant to do, since the base class signature also changed.
2. **Delete it** — one line of `git rm`, defensible if `examples/multimodal/run_text_generation.py` already
covers the VLM path.
3. **Leave it and mark it unmaintained** in a comment or in the docs, so the next reader does not spend time
on it.

Say which you would prefer and I will open the PR.

One more thing in the same file, only relevant if you choose option 1: line 138 is
`for prompt, generation in zip([prompt], resp_sentences):`, where the loop variable shadows the iterable it
is built from, so the loop body sees a value that changes underneath it on the next iteration.

Found by reading the code at `a97b0e2031`; the file was not executed (it cannot be). No performance claim is
involved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.