keymanapp / keymanapp/keyman

feat(developer,web): decrease model file size by reducing repetition 💾

Open
#7,006 0 comments 0 reactions 1 assignee Claimed by @mcdurdin View on GitHub
developer/ developer/compilers/ epic-model-encoding feat web/ web/predictive-text/
Dominant language
Pascal
Stars
534
Forks
143
Avg merge
2d 10h
Merged PRs (30d)
113

Description

**Is your feature request related to a problem? Please describe.**

Some models can be very large. We have a substantial amount of repetition in the compiled file which we could easily improve, AFAICT. Suggestions per this snippet here:

```js
var x = {"children":{"﷐":{"type":"leaf","weight":71058,"entries":[{"key":"եւ","weight":71058,"content":"և"},
{"key":"եւ","weight":71058,"content":"եւ"}]},"ս":{"type":"leaf","weight":1609,
"entries":[{"key":"եւս","weight":1609,"content":"ևս"},{"key":"եւս","weight":1609,
"content":"եւս"}]},"՛":{"type":"leaf","weight":842,"entries":[{"key":"եւ՛","weight":842,"content":"և՛"}]},
"ե":{"type":"internal","weight":150,"values":["թ"],"children":{"թ":{"type":"leaf","weight":150,
"entries":[{"key":"եւեթ","weight":150,"content":"ևեթ"},{"key":"եւեթ","weight":150,"content":"եւեթ"}]}}},
"՜":{"type":"leaf","weight":23,"entries":[{"key":"եւ՜","weight":23,"content":"և՜"}]}}};
```

1. Reduce field names and remove quotes (this is no longer JSON after all):
* `"children"` -> `c`
* `"type"` -> `t`
* `"weight"` -> `w`
* `"entries"` -> `e`
* `"key"` -> `k`
* `"content"` -> `C`
2. Simplify `"type"` (now `t`) field:
* `"leaf"` -> omit the field
* `"internal"` -> `1`
3. Reduce single-value arrays:
* `v:["?"]` -> v:"?"

With this, that snippet reduces to:

```js
var x = {c:{"﷐":{w:71058,e:[{k:"եւ",w:71058,c:"և"},{k:"եւ",w:71058,c:"եւ"}]},
"ս":{w:1609,e:[{k:"եւս",w:1609,c:"ևս"},{k:"եւս",w:1609,c:"եւս"}]},"՛":{w:842,
e:[{k:"եւ՛",w:842,c:"և՛"}]},"ե":{t:1,w:150,v:"թ",c:{"թ":{w:150,e:[{k:"եւեթ",w:150,
c:"ևեթ"},{k:"եւեթ",w:150,c:"եւեթ"}]}}},"՜":{w:23,e:[{k:"եւ՜",w:23,c:"և՜"}]}}};
```

In one example dotland.hy.armenian model (with 1-value words removed), these simple changes reduce the file size from 13,982KB to 7,660KB.

Ref: https://community.software.sil.org/t/text-suggestion-not-working-in-some-cases/6490

---

**KeymanWeb:**
- KeymanWeb Version: 15.0.267

**Keyman Developer:**
- Keyman Developer version: 15.0.267

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.