fohrloop / fohrloop/granite-tools

Make the bigram score scale fixed

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

One problem with fitting the bigram scores (using bigram score ratios and `granite-bigram-scores-fit`) is that sometimes the scale can vary greatly. The highest bigram score can become 6.0 or 12.0. Here is an example where I simply removed the unigram Q and score ratios associated to it from the bigram score ratio file (the highest score moves from about 7.8 to about 9.6):

![Image](https://github.com/user-attachments/assets/ac5bb2d6-9e59-4b12-b6d4-423a60826ea6)

As a fix, when calculating the bigram scores get the score ratio between the first and last ranked ngram in bigram score ratio file. For example (this is same as `ZQ/D` on the left side):

```yml
- ref: "K"
ngram: "/P"
score_ratio: 4.5
```
and scale all the calculated scores in a way that the resulting scores satisfy this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.