tools/run_vlm_text_generation.py cannot be imported: it uses megatron.inference.text_generation, removed by the legacy-inference deprecation
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
`tools/run_vlm_text_generation.py` cannot be imported. Lines 19-20 are
```python
from megatron.inference.text_generation.api import generate_and_post_process
from megatron.inference.text_generation.forward_step import ForwardStep
```
but `megatron/inference/` now contains only `__init__.py` and `utils.py` — the `text_generation`
subpackage was removed by `4d5135359` ("ADLR/megatron-lm!3818 - Deprecate legacy inference"). Both imports
are at module top level, so the script raises `ModuleNotFoundError` before it parses an argument.
It is also broken in a second, independent way that predates that deletion, which is the part that suggests
nobody has run it in a long time: `VLMForwardStep.__init__` calls
`super().__init__(model, max_batch_size, max_sequence_length)` (line 170), while at the last commit where
the import resolved the base constructor was `def __init__(self, model, inference_context)`. So restoring
the import alone would not make the file work.
Tagging @NVIDIA/mcore-oncall.
**Steps/Code to reproduce bug**
No GPU, no data, no checkpoint needed:
```bash
python -c "import tools.run_vlm_text_generation"
# ModuleNotFoundError: No module named 'megatron.inference.text_generation'
git ls-tree HEAD megatron/inference/
# only __init__.py and utils.py
```
**Expected behavior**
Either the tool runs, or it is not shipped. Right now it is shipped and cannot be imported.
**Additional context**
I am not sending a PR because the right fix is a decision I cannot make for you. Three options, and I am
happy to do any of them:
1. **Port it** to `megatron.core.inference` — the largest option, and it needs someone who knows what the
tool is meant to do, since the base class signature also changed.
2. **Delete it** — one line of `git rm`, defensible if `examples/multimodal/run_text_generation.py` already
covers the VLM path.
3. **Leave it and mark it unmaintained** in a comment or in the docs, so the next reader does not spend time
on it.
Say which you would prefer and I will open the PR.
One more thing in the same file, only relevant if you choose option 1: line 138 is
`for prompt, generation in zip([prompt], resp_sentences):`, where the loop variable shadows the iterable it
is built from, so the loop body sees a value that changes underneath it on the next iteration.
Found by reading the code at `a97b0e2031`; the file was not executed (it cannot be). No performance claim is
involved.
Contributor guide
Assessment
This issue has not been assessed yet.