alan-turing-institute / alan-turing-institute/deepsensor
Implement `TaskLoader.save` when instantiated with xarray/pandas objects
- Dominant language
- Python
- Stars
- 129
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
Currently, the `TaskLoader` can only be `.save`d when it has been `__init__`ed with _filepaths_ in its `context` and `target` entries, _not xarray/pandas data_. However, this forces the user to have to save their normalised xarray/pandas data, when they might not actually care that much about where that data lives. For example:
```
data_processor = DataProcessor(...)
da_normalised = data_processor(da_raw)
data_processor.save("folder")
da_normalised.save("fpath.nc") # We could potentially bypass this...
task_loader = TaskLoader(context="fpath.nc", target="fpath.nc") # By instead initialising with raw xarray/pandas here...
task_loader.save("folder") # And then `.save` would save the raw data objects alongside the TaskLoader config
```
We could instead initialise the `TaskLoader` in the typical way with raw xarray/pandas objects (which is more intuitive than fpaths), and then when saving the `TaskLoader` it will save those variables alongside the JSON config (with context/target file paths populated).
This FR should only be implemented after https://github.com/tom-andersson/deepsensor/issues/82 is closed. We don't want to save the same data multiple times just because it appears multiple times in the `context` and/or `target entries. So we'll want to leverage whatever internal `TaskLoader` data structure is added to close https://github.com/tom-andersson/deepsensor/issues/82.
### Basic Example
If this feature were implemented, we'd be able to do:
```
data_processor = DataProcessor(...)
da_normalised = data_processor(da_raw)
data_processor.save("folder")
task_loader = TaskLoader(context=da_normalised, target=da_normalised)
task_loader.save("folder") # This saves the context and target data as NetCDF/CSV in `"folder"`
```
See comment above - we will not want to save two NetCDF files in this case, because they are the same objects.
### Drawbacks
The user might not realise that `task_loader.save` will save data to disk, which is especially risky with very large NetCDF data and when disk space is limited. We'll need to be clear in the documentation that this is what is happening under the hood.
### Unresolved questions
_No response_
### Implementation PR
_No response_
### Reference Issues
_No response_
Contributor guide
Research direction
Start by reading issue #82 and the TaskLoader initialization and save entry points described here, since this feature depends on the internal data structure added there. Verify how repeated context and target objects would be identified and how NetCDF/CSV outputs and the JSON configuration should be written. Done means raw xarray/pandas objects can be passed to TaskLoader, saved alongside its configuration without duplicate files, and the disk-writing behavior is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100