mars-project / mars-project/mars

Loading and saving external data

Open
#105 3 comments 0 reactions 0 assignees View on GitHub
mod: tensor type: feature
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

This might be a silly question, but how do you load and save data with mars?

With dask.array, the main APIs are `dask.array.from_array` and `dask.array.store`. These allow for a great deal of flexibility, users can pass in arbitrary Python objects that support the `__getitem__` and `__setitem__` protocols.

I am intrigued about the idea of testing with mars with [xarray](http://xarray.pydata.org), but these APIs would be pretty essential for us. In xarray, we use these to support IO to various file formats (e.g., HDF5, netCDF, zarr).

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Mars's existing data-loading and saving entry points, then compare them with dask.array.from_array and dask.array.store, which the issue identifies as the reference APIs. The work is complete when Mars has a defined approach for loading and saving external data, including objects supporting __getitem__ and __setitem__ and the intended xarray-oriented use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.