cytomining / cytomining/CytoTable

Enable automatic `local_cache_dir` specification for cloud-based SQLite data sources

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

> Why isn't this default for CytoTable sqlite files? Is it too difficult or niche to implement?

_Originally posted by @gwaybio in https://github.com/cytomining/CytoTable/pull/138#discussion_r1445104349_

This issue focuses on enabling automatic `local_cache_dir` specification for `cloudpathlib` in order to effectively make use of cloud-based SQLite data sources within CytoTable. As-is, a user must manually specify this location when working with large SQLite data sources from the cloud for CytoTable. It may be possible to make use of the `target_path` and a subdirectory (cleaning up after processing is completed) as a default / automatic storage location for this.

Related: documentation updates with rough estimates about `target_path` data storage size expectations. For example, a user should be able to understand "How much data storage space will I need to run CytoTable with this data?".

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by tracing how CytoTable passes cloud-based SQLite sources to cloudpathlib and how target_path is handled. Define the expected automatic local_cache_dir lifecycle, including cleanup, and update documentation with storage-size guidance; done means cloud SQLite sources work without manual cache configuration and the storage expectations are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
cloud, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.