nextflow-io / nextflow-io/nextflow
Embedded cache DB alternative
Nobody has claimed this yet.
- #6255 by @pditommaso — closed without merging
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Nextflow tasks metadata is stored into a local embedded key-value database based on LevelDB.
This provides good performance, however, the LevelDB store has some stability issues on specific hardware/file systems causing and represent blocking factor for those users. See for example: https://github.com/nextflow-io/nextflow/issues/2377, https://github.com/nextflow-io/nextflow/issues/403, https://github.com/nextflow-io/nextflow/issues/351 and https://github.com/nextflow-io/nextflow/issues/309
The goal of this issue is to explore the use of lmdbjava as alternative storage for nextflow tasks metadata
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the LevelDB-based task-metadata storage and the related issues #2377, #403, #351, and #309, then inspect closed PR #6255 for prior exploration. Done means reaching a clear decision on the lmdbjava alternative and, if pursued, validating it against the reported stability problems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy, java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100