clab / clab/fast_align

Not able to train model for chinese

Open
#54 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
768
Forks
164
PR merge metrics
No merged PRs in 30d

Description

While training a chinese corpus, my model doesn't seems to train. It is giving the following heuristic at the end of each iteration.

``` log_e likelihood: nan
log_2 likelihood: nan
cross entropy: -nan
perplexity: -nan
posterior p0: 0
posterior al-feat: 0
size counts: 33102
```

Some lines from training corpus:
```1 . 至 4 . 备注 ||| 1. through 4. General Remarks
3月 6日 , 安理会 举行 了 一 次 非公开 会议 ( 第4286 次 会议 ) 南斯拉夫 联盟 共和国 总理 佐兰 · 日日奇 参加 了 会议 。 ||| On 6 March, the Council held a private meeting (4286th) with the participation of the Prime Minister of the Federal Republic of Yugoslavia, Zoran Žižić.
为 了 实现 这个 目标 , 实现 千 年 发展 目标 , 我们 认为 拥有 资源 的 国 家 必须 努力 提供 与 此 挑战 相 适宜 的 资金 。 ||| To achieve that objective and to attain the Millennium Development Goals, we believe that the countries possessing the resources must make a financial effort commensurate with the challenge.
㈡ 在 政府 间 论坛 上 对 咨询 服务 表示 满意 的 机构 的 数目 ||| (ii) Number of institutions expressing satisfaction with advisory services in intergovernmental forums
本 文件 的 增编 详细 介绍 了 中心 在 1995 - 1996年 发挥 的 作用 。 ||| Details of the Centre's role during the period 1995-1996 is provided in an addendum to the present document.
```

I'm using thulac for tokenizing chinese corpus.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the training failure using the supplied Chinese corpus examples and THULAC-tokenized input. Compare the NaN likelihood output with training on a known-good corpus, then trace the iteration where the heuristic values become NaN. Done means identifying a reproducible cause and confirming that the affected corpus trains without NaN values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.