dtolnay / dtolnay/proc-macro2

Consider recalibrating how bits are divided in Span

Open
#413 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
934
Forks
135
PR merge metrics
No merged PRs in 30d

Description

Currently fallback spans store a pair of 32-bit low and high character indices.

https://github.com/dtolnay/proc-macro2/blob/fecb02df0e2966c7eb39e020dbf650fa8bafd0c5/src/fallback.rs#L491-L496

A span in which `lo > hi` is malformed, so right off the bat, approximately half of possible Span bit patterns are wasted.

Separately, tokens are usually small compared to the total amount of input parsed by a thread. If we switch to storing `lo` and `hi - lo` instead of `lo` and `hi`, then an even split of 32 bits each may not be the wisest allocation. For example, we could decide to give 36 bits to `lo` (supporting 64 GB input size) and 28 bits to `hi - lo` (limiting token size to 256 MB). Or some other uneven split.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.