dtolnay / dtolnay/unicode-ident

Generalize to offset leaf table address for XID_Start vs XID_Continue

Open
#47 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
110
Forks
29
PR merge metrics
No merged PRs in 30d

Description

Currently this crate uses the same LEAF table address within both `is_xid_start` and `is_xid_continue`. This imposes a hard limit of 128.5 compressed chunks that can be accommodated because the maximum possible half-chunk index of 255 refers to a chunk beginning at offset 127.5.

It appears likely that we will exceed 128.5 chunks within a few upcoming Unicode releases, based on the rate at which Unicode has been introducing identifier characters:

| Unicode version | release date | compressed chunks |
|---|---|---|
| 6.1.0 | 2012.01.31 | 78 |
| 6.2.0 | 2012.09.26 | 78 |
| 6.3.0 | 2013.09.30 | 78 |
| 7.0.0 | 2014.06.16 | 91.5 |
| 8.0.0 | 2015.06.17 | 95.5 |
| 9.0.0 | 2016.06.21 | 100 |
| 10.0.0 | 2017.06.20 | 102 |
| 11.0.0 | 2018.06.05 | 107 |
| 12.0.0 | 2019.03.05 | 109.5 |
| 12.1.0 | 2019.05.07 | 109.5 |
| 13.0.0 | 2020.03.10 | 112 |
| 14.0.0 | 2021.09.14 | 113.5 |
| 15.0.0 | 2022.09.13 | 117.5 |
| 15.1.0 | 2023.09.12 | 118.5 |
| 16.0.0 | 2024.09.10 | 121.5 |
| 17.0.0 | 2025.09.09 | 122 |

Fortunately there is relatively little overlap between the chunks reachable from `is_xid_start` and chunks reachable from `is_xid_continue`. Most of the LEAF table is only used by one or the other of these. Specifically, only 70.5 chunks are reachable by `is_xid_start` and 75.5 are reachable by `is_xid_continue`; 20 are reachable by both. This suggests we actually have a lot of headroom by offsetting one of the tables in the following way, with no change in machine instructions (other than the relocation) and zero expected impact on performance:

LEAF_START:zeros


chunks for only XID_Start


LEAF_CONTINUE:zeros

chunks for both



chunks for only XID_Continue

@yongqli perhaps you would be interested to work on implementing this.

It can be tested by synthesizing some extra identifier characters like this:

```diff
diff --git a/generate/src/parse.rs b/generate/src/parse.rs
@@ -21,7 +21,7 @@ impl Properties {
}

pub fn is_xid_continue(&self, ch: char) -> bool {
- self.xid_continue.contains(&(ch as u32))
+ self.xid_continue.contains(&(ch as u32)) || (ch as u32).is_multiple_of(74699)
}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.