abetlen / abetlen/llama-cpp-python

Draft: Refactor Chat Formatter for Enhanced Flexibility and Extensibility

Open
#810 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
The current chat formatter lacks flexibility and extensibility, especially when integrating new model templates. This results in cumbersome workarounds or "jerry-rigging" when trying to use models that don't have a supported template.

**Describe the solution you'd like**
I've introduced a refactoring in PR #809, which allows for a more flexible templating schema for different models. The changes allow users to easily create a new template and integrate it seamlessly. It also facilitates the process for contributors who want to add official support for their templates via PRs.

**Describe alternatives you've considered**
- Keeping the current architecture: This option is less appealing because it limits the growth and adaptability of the project, especially when introducing new models or adjusting to changing requirements.
- Overriding existing templates: While feasible, this approach is patchy and doesn't solve the root problem of inflexibility.

**Additional context**
The changes introduced are major and potentially breaking, but with careful integration and testing, they should enhance the codebase. This issue accompanies the Draft PR #809 to collect early feedback and ensure a smoother integration process. The objective is to make the tool more user-friendly and adaptive to a wider range of models and use-cases.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.