arviz-devs / arviz-devs/arviz

InferenceData.to_netcdf and Arviz.from_netcdf be able to take file objects or buffers

Open
#1,237 5 comments 4 reactions 0 assignees View on GitHub
Enhancement Help Wanted
Dominant language
TeX
Stars
1.9k
Forks
507
Avg merge
2d 20h
Merged PRs (30d)
3

Description

Most libraries that allow writing or reading files also allow reading and writing to a python file object or an IO file-like object (eg. BytesIO). Currently Arviz requires writing these files to disk which creates an awkward dance if one simply wants to upload to s3, for instance. This may not seem like a big deal, but the files can be large, you have to remember to delete them off disk after you're done with them, etc.

Currently, to upload to S3, I have to do the following:
```
with tempfile.NamedTemporaryFile() as fp:
inference_data.to_netcdf(fp.name)
fp.seek(0)
s3.Bucket("bucket-name").upload_fileobj(
fp, "my-s3-file-key"
)
```

Would be much easier to do this (without writing to disk)

```
with BytesIO() as buffer:
inference_data.to_netcdf(buffer)
s3.Bucket("bucket-name").upload_filobj(buffer, "my-s3-file-key")
```

Even better, I'd suggest using [s3fs](https://s3fs.readthedocs.io/en/latest/) which is what Pandas does and support s3:// "protocol" so:

```
inference_data.to_netcdf("s3://bucket-name/my-s3-file-key")
```

However this last suggestion is not specifically related to the more important ability to write/read from a buffer or file object.

Contributor guide

Open the contributing guide

Research direction

Start at the InferenceData.to_netcdf and Arviz.from_netcdf entry points named in the issue, and trace how they currently handle filesystem paths. Define completion as supporting Python file objects or buffers for both writing and reading without requiring temporary disk files; the optional s3fs protocol is separate from the core request.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.