New SIMD arch
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 19
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
The idea is to keep the two-speed architecture, to still avoid PCLMULQDQ but to switch from one branch per structural char to one branch per chunk.
- POC
- bench
- SWAR version
- avx2 etc. since it becomes potentially viable
- try to make the parsing more robust to quotes that are not in beginning position
The pseudocode idea is the following:
per chunk, compute the newline + quote mask and the delimiter one
we mask the delimiter one with the structural one (we need to find the first set, then merge masks from here)
we process the delimiter mask in a loop
we get first structural from the other mask and we branch accordingly
either continue
either newline
either quote
If we have quote, we consume remaining bits of the mask in hope of finding a next quote, in which case we go to Quote instead of Quoted, so we don't lose work.
We might even want to attempt returning in Unquoted in a single chunk
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the current two-speed parser and its benchmark setup, then compare the issue's per-chunk mask approach with the existing structural-character branching. Work through the POC, benchmark, SWAR, and AVX2 checklist, and verify that quote handling also covers quotes that are not at the beginning of a chunk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100