huggingface / huggingface/transformers

Number-specific tokenization changes

Open
#15,485 0 comments 0 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 9h
Merged PRs (30d)
281

Description

# 🌟 New model addition

## Model description

I wanted to contribute a bunch of number-specific LMs proposed in recent work. Most of these are not architecture changes but simple tokenization tricks such as converting a number `329` to `3.29e2` (scientific; [Zhang et al. 2020](https://aclanthology.org/2020.findings-emnlp.439/)) or `3 2 9` (digit splitting; [Nogueira et al. 2021](https://arxiv.org/abs/2102.13019)) or `e2` (exponent only; [Spokoyny et al. 2020](https://aclanthology.org/2020.emnlp-main.385/) and [Thawani et al. 2021](https://aclanthology.org/2021.emnlp-main.557/)). The motivation is that several industrial applications require number-heavy NLP but struggle with existing models.

I discussed a specific way to do this on team slack with @SaulLu for, say, NumBERT (scientific notation) which involves adding a new model (tokenizer-only) and uploading the pretrained weights to the hub. I wanted to open a broader discussion here about more such number-tokenizer-only methods, some of which may not even have pretrained weights. The hope would be to make some abstract intervention (perhaps at the tokenizer level) to let the user configure GPT or BERT tokenizer as `number_tokenizer=exponent`.

But perhaps clubbing methods from different papers into one model/tokenizer is against HF's philosophy? If so, I could proceed with trying to simply incorporate them as individual models - is it fine if some of them do not have pretrained weights available?

## Open source status

* [X] the model implementation is available: [T5 fine-tuned on arithmetic, based on HF transformers](https://github.com/castorini/transformers-arithmetic) and [NumBERT/scientific, based on google-bert original code](https://github.com/google-research/google-research/tree/master/numbert)
* [X] the model weights are available: [BERT pretrained with scientific notation](https://console.cloud.google.com/storage/browser/gresearch/numbert)
* [X] who are the authors: @spokoyny @XikunZhang @DeepakRamachandran @iftenney @yanaiela @rodrigonogueira4 @lintool

Contributor guide

Open the contributing guide

Research direction

The issue names no Transformers file, test, or entry point. Start by reviewing the linked transformers-arithmetic and NumBERT implementations, then determine whether configurable tokenizer behavior or separate model integrations fit the project. Done would require an agreed scope, implementation plan, and clear requirements for pretrained weights.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.