Unknown "words" in text.txt with Updating the language model
- Langage dominant
- Jupyter Notebook
- Étoiles
- 15.1k
- Forks
- 1.8k
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
I want to build a new grammar with a text.txt, all the commands are ok but the last one:
farcompilestrings --fst_type=compact --symbols=words.txt --keep_symbols text.txt | \
ngramcount | ngrammake | \
fstconvert --fst_type=ngram > Gr.new.fst
- If all the words in text.txt are in the words.txt => OK
- If there are "new words" in the text.txt (unknown words) => there are errors like:
`FATAL: FarCompileStrings: Compiling string number 2 in file text.txt failed with token_type = symbol and entry_type = line`
I read the -help and use the new command:
farcompilestrings --fst_type=compact --symbols=words.txt --unknown_symbol="" --keep_symbols text.txt | ngramcount | ngrammake | fstconvert --fst_type=ngram > Gr.new.fst
Another error raised:
`FATAL: FarCompileStrings: Label "-1" missing from symbol table: words.txt
FATAL: STListReader::STListReader: Wrong file type: `
I know that: "You can not introduce new words this way, that is something we will cover later.", but
Are there any ways to deal with "new words" in a big text?
Help me, plz!
Thanks in advance!
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.