Not able to train model for chinese
- Dominant language
- C++
- Stars
- 768
- Forks
- 164
- PR merge metrics
- No merged PRs in 30d
Description
While training a chinese corpus, my model doesn't seems to train. It is giving the following heuristic at the end of each iteration.
``` log_e likelihood: nan
log_2 likelihood: nan
cross entropy: -nan
perplexity: -nan
posterior p0: 0
posterior al-feat: 0
size counts: 33102
```
Some lines from training corpus:
```1 . 至 4 . 备注 ||| 1. through 4. General Remarks
3月 6日 , 安理会 举行 了 一 次 非公开 会议 ( 第4286 次 会议 ) 南斯拉夫 联盟 共和国 总理 佐兰 · 日日奇 参加 了 会议 。 ||| On 6 March, the Council held a private meeting (4286th) with the participation of the Prime Minister of the Federal Republic of Yugoslavia, Zoran Žižić.
为 了 实现 这个 目标 , 实现 千 年 发展 目标 , 我们 认为 拥有 资源 的 国 家 必须 努力 提供 与 此 挑战 相 适宜 的 资金 。 ||| To achieve that objective and to attain the Millennium Development Goals, we believe that the countries possessing the resources must make a financial effort commensurate with the challenge.
㈡ 在 政府 间 论坛 上 对 咨询 服务 表示 满意 的 机构 的 数目 ||| (ii) Number of institutions expressing satisfaction with advisory services in intergovernmental forums
本 文件 的 增编 详细 介绍 了 中心 在 1995 - 1996年 发挥 的 作用 。 ||| Details of the Centre's role during the period 1995-1996 is provided in an addendum to the present document.
```
I'm using thulac for tokenizing chinese corpus.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the training failure using the supplied Chinese corpus examples and THULAC-tokenized input. Compare the NaN likelihood output with training on a known-good corpus, then trace the iteration where the heuristic values become NaN. Done means identifying a reproducible cause and confirming that the affected corpus trains without NaN values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100