High frequency token segmented into letter sequence when input is a tsv file
- Dominant language
- C++
- Stars
- 12.1k
- Forks
- 1.4k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 19
Description
Hi,
Currently I need to train a SP model. Since the input files are too large (~80G, 500M lines), I set up the program to use tsv file instead of raw text files as input. The format of input file is simply "word\tfrequency". I trained the model using unigram-lm. After training, I found some short but high frequency words are tokenized into character sequences. Below is an example:
```
echo "i am going to the park on monday" | spm_encode --model=sp-lm.model
▁ i ▁ a m ▁going ▁to ▁the ▁park ▁ o n ▁monday
```
However the frequency is:
```
i 144250667
am 5376197
on 79402723
park 1233890
```
The result is somewhat counterintuitive. Did I make any mistake or is it the algorithm indeed does or something else? The word freq file was generated from an untokenized dataset, has ~60M lines. My training command is
```bash
spm_train \
--input=counts \
--input_format=tsv \
--model_prefix=prefix \
--vocab_size=48000 \
--character_coverage=0.9999 \
--num_threads=50 \
--max_sentence_length=2048 \
--normalization_rule_name=identity \
--unk_surface="" \
--train_extremely_large_corpus \
--user_defined_symbols="," \
--byte_fallback \
--split_digits
```
Contributor guide
Research direction
Start by reproducing the reported output with the supplied spm_train TSV command and the spm_encode example. Read the unigram-lm training and TSV frequency-handling paths to determine whether high-frequency entries are treated as expected; done means explaining the segmentation or identifying a reproducible bug and covering it with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100