ngless-toolkit / ngless-toolkit/ngless
Allow collect()'ing when all processing is complete
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 152
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
Scenario:
- 12 samples are being processed using the
parallelmachinerylock1()andcollect(). - 10 samples complete and 2 fail.
- The 2 failing samples are considered bad and are excluded from the sample file.
At this point re-running ngless has no effect since all work is complete however the merged output from collect() was never generated.
collect() can also fail to occur in rare cases where the last two samples finish almost simultaneously or filesystem lag prevents the last two processes from seeing all samples as complete.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the parallel machinery around lock1() and collect(), focusing on completion detection when failed samples are excluded and when the final processes finish together. Reproduce the reported scenario, including filesystem lag if possible. Done means collect() reliably generates the merged output after all processing is complete, including simultaneous completion and failed samples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100