composefs / composefs/composefs-rs
Potential pulling / syncing optimization
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 93
- Forks
- 30
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 13
Description
composefs-rs used to fsync each file written when pulling which was costly.
This got changed to a global syncfs call in https://github.com/composefs/composefs-rs/commit/c3677f1a3599be0015db6798b60c5bdfa3e43d7e.
We could potentially do better by following a logic similar to what's done in ostree:
- When pulling, link all the files to be written into a tmp folder
- Then fsync the entire folder once ready to move them into the composefs repo
- Then complete the repo import
- We would have a tmp directory per pull operation to enable parallel pulls
Some notes from Pragyan:
Adding some stats related to fsync:
For 37857 objects, we call fsync 4 times (I’m not sure at what point we sync everything)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the referenced commit and the ostree synchronization logic, then trace the pull and repository-import path in composefs-rs. The work is done when pulling uses a per-operation temporary directory, synchronizes the staged files before import, supports parallel pulls, and preserves repository correctness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100