huggingface / huggingface/lighteval

[BUG] TransformersModel.from_model provides an incompletely initialized object compared to __init__

Open
#897 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When creating a `TransformersModel` instance using the `from_model` classmethod, the resulting object is missing several attributes (e.g.continuous_batching) that are normally initialized by the standard `__init__ `constructor. This leads to AttributeError exceptions during the evaluation pipeline, making the `from_model` method unreliable for integrating pre-loaded models.

## To Reproduce
1, In an evaluation script, pre-load a model from transformers.
2. Instantiate the lighteval model wrapper using this pre-loaded model: model = TransformersModel.from_model(model=hf_model, ...)
3. Pass this model instance to a lighteval.pipeline.Pipeline.
4. Run pipeline.evaluate().
5. The pipeline will crash with an AttributeError when an internal method attempts to access an attribute that was not initialized by from_model (for example, `AttributeError: 'TransformersModel' object has no attribute 'continuous_batching')`.

## Expected behavior
An object created via `TransformersModel.from_model` should be fully and correctly initialized, possessing the exact same set of attributes as an object created via the standard `TransformersModel(config=...) ` constructor. It should be a drop-in replacement that works reliably within the pipeline.

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.