dtolnay / dtolnay/unicode-ident
Generalize to offset leaf table address for XID_Start vs XID_Continue
- Dominant language
- Rust
- Stars
- 110
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
Currently this crate uses the same LEAF table address within both `is_xid_start` and `is_xid_continue`. This imposes a hard limit of 128.5 compressed chunks that can be accommodated because the maximum possible half-chunk index of 255 refers to a chunk beginning at offset 127.5.
It appears likely that we will exceed 128.5 chunks within a few upcoming Unicode releases, based on the rate at which Unicode has been introducing identifier characters:
| Unicode version | release date | compressed chunks |
|---|---|---|
| 6.1.0 | 2012.01.31 | 78 |
| 6.2.0 | 2012.09.26 | 78 |
| 6.3.0 | 2013.09.30 | 78 |
| 7.0.0 | 2014.06.16 | 91.5 |
| 8.0.0 | 2015.06.17 | 95.5 |
| 9.0.0 | 2016.06.21 | 100 |
| 10.0.0 | 2017.06.20 | 102 |
| 11.0.0 | 2018.06.05 | 107 |
| 12.0.0 | 2019.03.05 | 109.5 |
| 12.1.0 | 2019.05.07 | 109.5 |
| 13.0.0 | 2020.03.10 | 112 |
| 14.0.0 | 2021.09.14 | 113.5 |
| 15.0.0 | 2022.09.13 | 117.5 |
| 15.1.0 | 2023.09.12 | 118.5 |
| 16.0.0 | 2024.09.10 | 121.5 |
| 17.0.0 | 2025.09.09 | 122 |
Fortunately there is relatively little overlap between the chunks reachable from `is_xid_start` and chunks reachable from `is_xid_continue`. Most of the LEAF table is only used by one or the other of these. Specifically, only 70.5 chunks are reachable by `is_xid_start` and 75.5 are reachable by `is_xid_continue`; 20 are reachable by both. This suggests we actually have a lot of headroom by offsetting one of the tables in the following way, with no change in machine instructions (other than the relocation) and zero expected impact on performance:
LEAF_START:zeros
…
…
chunks for only XID_Start
…
…
LEAF_CONTINUE:zeros
…
chunks for both
…
…
…
chunks for only XID_Continue
…
…
@yongqli perhaps you would be interested to work on implementing this.
It can be tested by synthesizing some extra identifier characters like this:
```diff
diff --git a/generate/src/parse.rs b/generate/src/parse.rs
@@ -21,7 +21,7 @@ impl Properties {
}
pub fn is_xid_continue(&self, ch: char) -> bool {
- self.xid_continue.contains(&(ch as u32))
+ self.xid_continue.contains(&(ch as u32)) || (ch as u32).is_multiple_of(74699)
}
}
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.