GSTT-CSC / GSTT-CSC/MLOps

Option to cache XNAT data

Open
#134 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
23
Forks
6
Avg merge
3h 56m
Merged PRs (30d)
1

Description

Importing a project's data from XNAT for each run is a large fraction of each iteration of training runs. This is because on each run the data is downloaded from the server to memory and then discarded when the script exits.

We should implement a caching system so that the downloaded data is stored in an ordered way on the local disk.

The persistence of this data could be defined by.
- time: a downloaded dataset is cached for a set amount of time and will be deleted after it expires
- size: a cached dataset directory is defined and will not exceed a certain size, new data coming in will prompt the deletion of old data from the cache.

@GSTT-CSC/core tagging for visibility/input.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.