cytomining / cytomining/CytoTable
Enable automatic `local_cache_dir` specification for cloud-based SQLite data sources
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
> Why isn't this default for CytoTable sqlite files? Is it too difficult or niche to implement?
_Originally posted by @gwaybio in https://github.com/cytomining/CytoTable/pull/138#discussion_r1445104349_
This issue focuses on enabling automatic `local_cache_dir` specification for `cloudpathlib` in order to effectively make use of cloud-based SQLite data sources within CytoTable. As-is, a user must manually specify this location when working with large SQLite data sources from the cloud for CytoTable. It may be possible to make use of the `target_path` and a subdirectory (cleaning up after processing is completed) as a default / automatic storage location for this.
Related: documentation updates with rough estimates about `target_path` data storage size expectations. For example, a user should be able to understand "How much data storage space will I need to run CytoTable with this data?".
Contributor guide
Research direction
The issue names no files or tests; start by tracing how CytoTable passes cloud-based SQLite sources to cloudpathlib and how target_path is handled. Define the expected automatic local_cache_dir lifecycle, including cleanup, and update documentation with storage-size guidance; done means cloud SQLite sources work without manual cache configuration and the storage expectations are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- cloud, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100