[huggingface] NameError: name 'AutoConfig' is not defined in download_model()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
Bug
Running any HuggingFace benchmark model that goes through download_model()
fails with:
File "torchbenchmark/util/framework/huggingface/basic_configs.py", line 304, in download_model
config = eval(HUGGINGFACE_MODELS[model_name][2])
NameError: name 'AutoConfig' is not defined
Root Cause
basic_configs.py line 303:
exec(f"from transformers import {config_cls_name}") # line 303
config = eval(HUGGINGFACE_MODELS[model_name][2]) # line 304 — NameError
In Python 3, exec() without an explicit globals dict imports into the
local scope of the enclosing function. The subsequent eval() resolves
names against module globals, not local scope — so AutoConfig (or any
other imported class) is invisible to it.
This is a well-known Python 3 scoping behaviour:
https://docs.python.org/3/library/functions.html#exec
Fix
exec(f"from transformers import {config_cls_name}", globals())
config = eval(HUGGINGFACE_MODELS[model_name][2])
Passing globals() makes the imported name available to eval().
Affected Models
All HuggingFace models whose config constructor string in
HUGGINGFACE_MODELS uses a class that must be imported (e.g.
AutoConfig.from_pretrained(...), LlamaConfig(...), etc.).
Reproduction
# Minimal reproduction of the bug
def broken():
exec("from transformers import AutoConfig")
return eval('AutoConfig.from_pretrained("albert-base-v2")') # NameError
def fixed():
exec("from transformers import AutoConfig", globals())
return eval('AutoConfig.from_pretrained("albert-base-v2")') # works
Environment
- Python 3.11+
- Any recent transformers version
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in torchbenchmark/util/framework/huggingface/basic_configs.py around lines 303-304 and reproduce the failure through download_model(). Check the interaction between the dynamic import and eval(), then verify that affected HuggingFace models no longer raise NameError when their config constructor is evaluated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100