Review file IO model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 150
- Forks
- 23
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
We agreed that, during or after the Dask work in #182, we should review the cf approach to file IO.
The topic arose in discussions relating to the new append mode (#30) because files were being inadvertently left open leading to HDF errors and seg faults (e.g. see https://github.com/NCAS-CMS/cf-python/pull/213#issuecomment-847101986) and though this was fixed, we realised there may be improvements to be made regarding the current caching of open files.
Moreover we realised the current file IO model might not work with the parallelisation and lazy loading resulting from Dask-ification so that would probably compel a change in approach, which could be a tweak or could be a large-scale change.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with issue #182 and the append-mode context in #30, then review the current file-IO caching model and the linked discussion in PR #213. Done means agreeing on and implementing a file-IO approach that avoids the reported open-file problems and remains compatible with Dask parallelisation and lazy loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100