huggingface / huggingface/transformers

Feature: Tail Free Sampling

Open
#13,784 3 comments 0 reactions 0 assignees View on GitHub
WIP
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 9h
Merged PRs (30d)
281

Description

# 🚀 Feature request

I would like to have Huggingface implement Tail Free Sampling, or TFS for short, to the official repo. The original paper for this can be found here: https://trentbrick.github.io/Tail-Free-Sampling/#tail-free-sampling-algorithm

Implementation of this code can be found here: https://github.com/finetuneanon/transformers/blob/gpt-neo-localattention3-rp-b/src/transformers/generation_logits_process.py#L243-L284

## Motivation

At the moment KoboldAI uses Finetuneanon's version of Transformers for creating stories, however, this version has several changes made to the repository that make it impossible for me to use (eg: I use an aarch64 architecture with an integrated Nvidia chip, which is only supported by Huggingface's main branch). There are calls from the KoboldAI devs to migrate due to the integrated support of GPT-J in Huggingface, however, there are some features in Finetuneanon's Transformer which are not found in the main branch. Tail Free Sampling is one such feature.

## Your contribution

I would like to file a PR with the code, however, I do lack the knowledge on how to implement the required tests and checks. An example is available on Finetuneanon's branch but it will require cherry-picking the commit to the main branch.

Contributor guide

Open the contributing guide

Research direction

Start with the Tail-Free Sampling paper and the referenced implementation in src/transformers/generation_logits_process.py, then compare it with the current generation code. Identify the required test location and verify the implementation against the paper and reference behavior; done means the feature is integrated with appropriate tests and checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.