mars-project / mars-project/mars
Loading and saving external data
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
This might be a silly question, but how do you load and save data with mars?
With dask.array, the main APIs are `dask.array.from_array` and `dask.array.store`. These allow for a great deal of flexibility, users can pass in arbitrary Python objects that support the `__getitem__` and `__setitem__` protocols.
I am intrigued about the idea of testing with mars with [xarray](http://xarray.pydata.org), but these APIs would be pretty essential for us. In xarray, we use these to support IO to various file formats (e.g., HDF5, netCDF, zarr).
Contributor guide
Research direction
Start by reviewing Mars's existing data-loading and saving entry points, then compare them with dask.array.from_array and dask.array.store, which the issue identifies as the reference APIs. The work is complete when Mars has a defined approach for loading and saving external data, including objects supporting __getitem__ and __setitem__ and the intended xarray-oriented use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100