huggingface / huggingface/transformers
Feature: Tail Free Sampling
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 281
Description
# 🚀 Feature request
I would like to have Huggingface implement Tail Free Sampling, or TFS for short, to the official repo. The original paper for this can be found here: https://trentbrick.github.io/Tail-Free-Sampling/#tail-free-sampling-algorithm
Implementation of this code can be found here: https://github.com/finetuneanon/transformers/blob/gpt-neo-localattention3-rp-b/src/transformers/generation_logits_process.py#L243-L284
## Motivation
At the moment KoboldAI uses Finetuneanon's version of Transformers for creating stories, however, this version has several changes made to the repository that make it impossible for me to use (eg: I use an aarch64 architecture with an integrated Nvidia chip, which is only supported by Huggingface's main branch). There are calls from the KoboldAI devs to migrate due to the integrated support of GPT-J in Huggingface, however, there are some features in Finetuneanon's Transformer which are not found in the main branch. Tail Free Sampling is one such feature.
## Your contribution
I would like to file a PR with the code, however, I do lack the knowledge on how to implement the required tests and checks. An example is available on Finetuneanon's branch but it will require cherry-picking the commit to the main branch.
Contributor guide
Research direction
Start with the Tail-Free Sampling paper and the referenced implementation in src/transformers/generation_logits_process.py, then compare it with the current generation code. Identify the required test location and verify the implementation against the paper and reference behavior; done means the feature is integrated with appropriate tests and checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100