medialab / medialab/simd-csv

New SIMD arch

Open
#73 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
19
Forks
2
PR merge metrics
No merged PRs in 30d

Description

The idea is to keep the two-speed architecture, to still avoid PCLMULQDQ but to switch from one branch per structural char to one branch per chunk.

  • POC
  • bench
  • SWAR version
  • avx2 etc. since it becomes potentially viable
  • try to make the parsing more robust to quotes that are not in beginning position

The pseudocode idea is the following:

per chunk, compute the newline + quote mask and the delimiter one
we mask the delimiter one with the structural one (we need to find the first set, then merge masks from here)

we process the delimiter mask in a loop
we get first structural from the other mask and we branch accordingly
  either continue
  either newline
  either quote

If we have quote, we consume remaining bits of the mask in hope of finding a next quote, in which case we go to Quote instead of Quoted, so we don't lose work.
We might even want to attempt returning in Unquoted in a single chunk

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the current two-speed parser and its benchmark setup, then compare the issue's per-chunk mask approach with the existing structural-character branching. Work through the POC, benchmark, SWAR, and AVX2 checklist, and verify that quote handling also covers quotes that are not at the beginning of a chunk.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.