huggingface / huggingface/evaluate

perplexity.compute - Add args to pass model and tokenizer objects instead of `model_id`

Open
#491 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hello,
I am new to the project, so I hope my request is a sensible one.

It looks like when we use the compute method for Perplexity, we can only specify a model through its `model_id`, which is used also to load the tokenizer.

Is there any particular reason why it is not possible to pass as arguments to the method the model object and the tokenizer object themselves instead of the `model_id`? I think that could help, for example when the tokenizer has some user-added tokens or when the model to be used is fine-tuned (e.g., currently, a PeftModel could not be used, but passing the model object itself that should not be a limitation anymore).

Thanks for your answer and for your amazing work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.