pytorch / pytorch/benchmark

[huggingface] NameError: name 'AutoConfig' is not defined in download_model()

Open Beginner friendly
#2,703 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

Bug

Running any HuggingFace benchmark model that goes through download_model()
fails with:

File "torchbenchmark/util/framework/huggingface/basic_configs.py", line 304, in download_model
    config = eval(HUGGINGFACE_MODELS[model_name][2])
NameError: name 'AutoConfig' is not defined
Root Cause

basic_configs.py line 303:

exec(f"from transformers import {config_cls_name}")   # line 303
config = eval(HUGGINGFACE_MODELS[model_name][2])      # line 304 — NameError

In Python 3, exec() without an explicit globals dict imports into the
local scope of the enclosing function. The subsequent eval() resolves
names against module globals, not local scope — so AutoConfig (or any
other imported class) is invisible to it.

This is a well-known Python 3 scoping behaviour:
https://docs.python.org/3/library/functions.html#exec

Fix
exec(f"from transformers import {config_cls_name}", globals())
config = eval(HUGGINGFACE_MODELS[model_name][2])

Passing globals() makes the imported name available to eval().

Affected Models

All HuggingFace models whose config constructor string in
HUGGINGFACE_MODELS uses a class that must be imported (e.g.
AutoConfig.from_pretrained(...), LlamaConfig(...), etc.).

Reproduction
# Minimal reproduction of the bug
def broken():
    exec("from transformers import AutoConfig")
    return eval('AutoConfig.from_pretrained("albert-base-v2")')  # NameError

def fixed():
    exec("from transformers import AutoConfig", globals())
    return eval('AutoConfig.from_pretrained("albert-base-v2")')  # works
Environment
  • Python 3.11+
  • Any recent transformers version

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in torchbenchmark/util/framework/huggingface/basic_configs.py around lines 303-304 and reproduce the failure through download_model(). Check the interaction between the dynamic import and eval(), then verify that affected HuggingFace models no longer raise NameError when their config constructor is evaluated.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.