Language_ja has identical codepoint coverage to Script_japanese — could language files be unfiltered aggregations of the language's pages?
@garretrieger is already working on this.
Since Sep 2, 2026.
- Dominant language
- C++
- Stars
- 4
- Forks
- 2
- Avg merge
- 10m
- Merged PRs (30d)
- 1
Description
We dumped both Language_ja and Script_japanese with freq_data_to_sorted_codepoints: identical sets, 9,877 codepoints each, CJK-script only — no …, no ASCII beyond the space. For a language data file this seems off: real Japanese pages almost always contain ASCII letters and digits and Common-script symbols (※ © → ① …) — none of which are covered (fullwidth forms, by contrast, are). Since none of these have frequency data, the cost merger can't evaluate them and they end up segregated into their own invalidating table-keyed segment — so ordinary Japanese text plus a single symbol costs two sequential round trips. In a stratified crawl of ~1,900 of our production sites, 18.5% of Japanese font loads spanned segments this way and 46.9% of pages carried at least one such load — even with all printable ASCII already placed in the initial segment; without that, virtually every page would span.
Request: could Language_ja (and language files generally) be regenerated as a plain aggregation over the language-detected pages, without the script filter? The README describes the filter for Script_* files — which makes sense there — but the language files are where unfiltered data matters: the collection pipeline already counts page-level co-occurrence pairs, so the JA×symbol and JA×Latin pairs the merger would need are observed today and then dropped. This would also feed nicely into w3c/ift-encoder#261: the cross-set pairs that matter most for CJK (JA×Common/Latin) would simply become intra-set.
Meanwhile our workaround, for reference, has three layers: all printable ASCII in the initial segment (this alone is what keeps the span rate down to the 18.5% above), plus the ~20 most frequent symbols measured from our crawl, plus jump_ahead: 2 — the latter two eliminated all remaining 3-round sessions in a replay of ~3,700 real browsing sessions. It works, but every CJK deployer would have to rediscover it. If an updated Language_ja needs real-world validation for Japanese, happy to help: our simulator replays real sessions against a candidate segmentation and reports transferred bytes and round counts per session.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.