Suggestion: Streamlining Tokenization and Model Loader in Hugging Face Models with Transformers Pipelines

Hey Team 👋

I hope this message finds you well. I would like to propose a feature enhancement that could simplify the usage of tokenization in Hugging Face models and improve overall code maintainability.

**Proposal:**
Currently, when defining custom models, users often need to manually load tokenizers and set up tokenizer loading methods within their class definitions. This can lead to repetitive code and increased complexity, making it less intuitive for users.

I propose that we leverage the power of Transformers Pipelines to handle tokenization for Hugging Face models more efficiently. Specifically, instead of manually loading tokenizers within the custom model class, we can utilize the pipeline mechanism provided by Transformers to handle tokenization more elegantly.

**Example:**
For instance, consider the current model structure:

```python
Copy code
@RegisterModelClass("neo")
class SmallLocalNeo(BaseOnsiteLLM):
"""
Attributes:
model_uri (str): Hugging Face Endpoint for LLM
tokenizer (AutoTokenizer): Tokenizer from Transformer's library
model (LLM): The large language model

Methods:
model_loader: Loads the LLM into memory
tokenizer_loader: Loads the tokenizer into memory
generate: Generates a response from a given prompt with the loaded LLM and tokenizer
"""
model_uri = "EleutherAI/gpt-neo-1.3B"

def model_loader(self):
return GPTNeoForCausalLM.from_pretrained(self.model_uri)

def tokenizer_loader(self):
return AutoTokenizer.from_pretrained(self.model_uri)
```

We can simplify this structure by eliminating the tokenizer_loader method and letting users leverage the Transformers pipeline for tokenization, like this:

```python
Copy code
from transformers import pipeline

@RegisterModelClass("neo")
class SmallLocalNeo(BaseOnsiteLLM):
"""
Attributes:
model_uri (str): Hugging Face Endpoint for LLM
model (LLM): The large language model

Methods:
model_loader: Loads the LLM into memory
generate: Generates a response from a given prompt with the loaded LLM
"""
model_uri = "EleutherAI/gpt-neo-1.3B"

def model_loader(self):
return pipeline(self.model_uri)
```

**Benefits:**

`Simplicity`: Users no longer need to define tokenizer loading methods in custom model classes, simplifying the codebase and making it more intuitive.

`Consistency`: Users can adopt a consistent approach across all models by using the same pipeline mechanism for tokenization.

`Ease of Use`: Utilizing pipelines aligns with the "batteries included" philosophy of Hugging Face Transformers, making it easier for users to get started.

`Reduced Maintenance`: With tokenization handled by the pipeline, users are less likely to encounter tokenization-related issues when defining custom models.

Furthermore, this enhancement aligns with the ongoing discussion in [issue #194] regarding the creation of a generalized class for all HuggingFace models. The proposed change can significantly contribute to this goal by simplifying the process of adding new models to the library.

I would appreciate your thoughts and feedback on this suggestion. If there are any concerns or potential drawbacks that I may have missed, please feel free to discuss them. Thank you for considering this proposal, and I look forward to your response!!