alan-turing-institute / alan-turing-institute/deepsensor

Implement `TaskLoader.save` when instantiated with xarray/pandas objects

Open
#84 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
129
Forks
29
PR merge metrics
No merged PRs in 30d

Description

### Summary

Currently, the `TaskLoader` can only be `.save`d when it has been `__init__`ed with _filepaths_ in its `context` and `target` entries, _not xarray/pandas data_. However, this forces the user to have to save their normalised xarray/pandas data, when they might not actually care that much about where that data lives. For example:
```
data_processor = DataProcessor(...)
da_normalised = data_processor(da_raw)
data_processor.save("folder")
da_normalised.save("fpath.nc") # We could potentially bypass this...
task_loader = TaskLoader(context="fpath.nc", target="fpath.nc") # By instead initialising with raw xarray/pandas here...
task_loader.save("folder") # And then `.save` would save the raw data objects alongside the TaskLoader config
```

We could instead initialise the `TaskLoader` in the typical way with raw xarray/pandas objects (which is more intuitive than fpaths), and then when saving the `TaskLoader` it will save those variables alongside the JSON config (with context/target file paths populated).

This FR should only be implemented after https://github.com/tom-andersson/deepsensor/issues/82 is closed. We don't want to save the same data multiple times just because it appears multiple times in the `context` and/or `target entries. So we'll want to leverage whatever internal `TaskLoader` data structure is added to close https://github.com/tom-andersson/deepsensor/issues/82.

### Basic Example

If this feature were implemented, we'd be able to do:
```
data_processor = DataProcessor(...)
da_normalised = data_processor(da_raw)
data_processor.save("folder")
task_loader = TaskLoader(context=da_normalised, target=da_normalised)
task_loader.save("folder") # This saves the context and target data as NetCDF/CSV in `"folder"`
```

See comment above - we will not want to save two NetCDF files in this case, because they are the same objects.

### Drawbacks

The user might not realise that `task_loader.save` will save data to disk, which is especially risky with very large NetCDF data and when disk space is limited. We'll need to be clear in the documentation that this is what is happening under the hood.

### Unresolved questions

_No response_

### Implementation PR

_No response_

### Reference Issues

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #82 and the TaskLoader initialization and save entry points described here, since this feature depends on the internal data structure added there. Verify how repeated context and target objects would be identified and how NetCDF/CSV outputs and the JSON configuration should be written. Done means raw xarray/pandas objects can be passed to TaskLoader, saved alongside its configuration without duplicate files, and the disk-writing behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.