huggingface / huggingface/lighteval
[BUG] TransformersModel.from_model provides an incompletely initialized object compared to __init__
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When creating a `TransformersModel` instance using the `from_model` classmethod, the resulting object is missing several attributes (e.g.continuous_batching) that are normally initialized by the standard `__init__ `constructor. This leads to AttributeError exceptions during the evaluation pipeline, making the `from_model` method unreliable for integrating pre-loaded models.
## To Reproduce
1, In an evaluation script, pre-load a model from transformers.
2. Instantiate the lighteval model wrapper using this pre-loaded model: model = TransformersModel.from_model(model=hf_model, ...)
3. Pass this model instance to a lighteval.pipeline.Pipeline.
4. Run pipeline.evaluate().
5. The pipeline will crash with an AttributeError when an internal method attempts to access an attribute that was not initialized by from_model (for example, `AttributeError: 'TransformersModel' object has no attribute 'continuous_batching')`.
## Expected behavior
An object created via `TransformersModel.from_model` should be fully and correctly initialized, possessing the exact same set of attributes as an object created via the standard `TransformersModel(config=...) ` constructor. It should be a drop-in replacement that works reliably within the pipeline.
## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.