google / google/sentencepiece

High frequency token segmented into letter sequence when input is a tsv file

Open
#967 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
12.1k
Forks
1.4k
Avg merge
1d 22h
Merged PRs (30d)
19

Description

Hi,

Currently I need to train a SP model. Since the input files are too large (~80G, 500M lines), I set up the program to use tsv file instead of raw text files as input. The format of input file is simply "word\tfrequency". I trained the model using unigram-lm. After training, I found some short but high frequency words are tokenized into character sequences. Below is an example:

```
echo "i am going to the park on monday" | spm_encode --model=sp-lm.model
▁ i ▁ a m ▁going ▁to ▁the ▁park ▁ o n ▁monday
```

However the frequency is:
```
i 144250667
am 5376197
on 79402723
park 1233890
```

The result is somewhat counterintuitive. Did I make any mistake or is it the algorithm indeed does or something else? The word freq file was generated from an untokenized dataset, has ~60M lines. My training command is

```bash
spm_train \
--input=counts \
--input_format=tsv \
--model_prefix=prefix \
--vocab_size=48000 \
--character_coverage=0.9999 \
--num_threads=50 \
--max_sentence_length=2048 \
--normalization_rule_name=identity \
--unk_surface="" \
--train_extremely_large_corpus \
--user_defined_symbols="," \
--byte_fallback \
--split_digits
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported output with the supplied spm_train TSV command and the spm_encode example. Read the unigram-lm training and TSV frequency-handling paths to determine whether high-frequency entries are treated as expected; done means explaining the segmentation or identifying a reproducible bug and covering it with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.