shared cache: add shared-file support to AXL to parallelize copy
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 108
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
From Adam:
With the current version of AXL, we'd have a single thread on a single compute node copy the entire file from cache to the parallel file system. However, since we're now dealing with a single shared file, the size of the file scales up with the node count of the job. We probably will want to split up a file into regions and assign different portions of the file to be copied by different (multiple) processes. And that seems like it will require us to modify the AXL API so that we can provide an offset and length in addition to the source and destination file names.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the AXL API and its current file-copy entry point in the C codebase; the issue does not name specific files or tests. Determine how shared files are represented and how multiple processes could receive source and destination regions, then verify that the API supports offset and length parameters and parallel copying.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100