apertium / apertium/lttoolbox

Write a utility to assign weights to a compiled transducer based on a corpus

Open
#16 9 comments 0 reactions 0 assignees View on GitHub
enhancement weighting
Dominant language
C++
Stars
21
Forks
26
PR merge metrics
No merged PRs in 30d

Description

I imagine it will be called `lt-reweight`

It should have two arguments:

1) a binary lttoolbox file e.g. ` grn.automorf.bin `
2) a tagged corpus `grn.tagged`

```
$ lt-reweight grn.automorf.bin grn.tagged
```

Where grn.tagged looks like:

```
^Avañeʼẽ/avañeʼẽ$
^ha/ha$
^Guarani/guarani$
^ñeʼẽ/ñeʼẽ$
^ombohéra/o+mbohéra$
^hikuái/hikuái$
^umi/umi$
^Guaranikuéra/guarani+kuéra$
^pe/pe$
^ñeʼẽ/ñeʼẽ$
^teépe/tee+pe$
^./.$

^Guarani/guarani$
^haʼe/haʼe$
^peteĩva/peteĩ+va$
^umi/umi$
^teʼyikuéra/teʼyi+kuéra$
^Amérika-gua/Amérika+gua$
^ñeʼẽnguéra/ñeʼẽ+kuéra$
^apytépe/apytépe$
^hetave/heta+ve$
^iñeʼẽhárava/iñeʼẽhárava$
^,/,$
^oñemohendáva/o+je+mohenda+va$
^irundy/irundy$
^tetãnguéra/tetã+kuéra$
^iñambuévape/iñambuéva+pe$
^(/($
^Paraguái/Paraguái$
^,/,$
^Argentina/Argentina$
^,/,$
^Volívia/Volívia$
^ha/ha$
^Brasil/Brasil$
^)/)$
^./.$

^Avei/avei$
^,/,$
^haʼe/haʼe$
^ñoite/ñoite$
^ojehechakuaáva/o+je+hechakuaa+va$
^ñeʼẽ/ñeʼẽ$
^teéramo/tee+ramo$
^peteĩ/peteĩ$
^tetã/tetã$
^Ñembyamérika-guápe/Ñembyamérika+gua+pe$
^./.$

^Tupi/Tupi$
^ha/ha$
^guarani/guarani$
^ñeʼẽ/ñeʼẽ$
^aty/aty$
^guasu/guasu$
^rehegua/rehegua$

^,/,$
^oguereko/o+guereko$
^hetáichagua/hetáichagua$

^ñeʼẽnunga/ñeʼẽnunga$
^,/,$
^upéicharõ/upéicha+rõ$
^jepe/jepe$
^oĩ/oĩ$
^jekupyty/jekupyty$
^ijapytepekuéra/i+japyte+pe+kuéra$
^ha/ha$
^heta/heta$
^mbaʼépe/mbaʼe+pe$
^ojojogua/ojojogua$
^koʼã/koʼã$
^ñeʼẽnungakuéra/ñeʼẽnunga+kuéra$
^./.$

^Avañeʼẽ/avañeʼẽ$
^ha/ha$
^karaiñeʼẽ/karaiñeʼẽ$
^haʼe/haʼe$
^Paraguái/Paraguái$
^retãme/tetã+pe$
^ñeʼẽ/ñeʼẽ$
^tee/tee$
^ary/ary$
^1992/1992$
^guive/guive$
^./.$

^Japypateĩ/Japypateĩ$
^2006/2006$
^guive/guive$
^haʼe/haʼe$
^avei/avei$
^ñeʼẽ/ñeʼẽ$
^tee/tee$
^Mercosur-pe/Mercosur+pe$
^,/,$
^karaiñeʼẽ/karaiñeʼẽ$
^ha/ha$
^poytugañeʼẽ/poytugañeʼẽ$
^ykére/ykére$
^./.$

```

And the output of the analyser for e.g. `poytugañeʼẽ` is:

```
^poytugañeʼẽ/poytugañeʼẽ/a+poytugañeʼẽ/re+poytugañeʼẽ$^./.$
```

So, the analyses should be weighted

```
poytugañeʼẽ : poytugañeʼẽ = 1.0
poytugañeʼẽ : a+poytugañeʼẽ = 0.0
poytugañeʼẽ : re+poytugañeʼẽ = 0.0
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.