mars-project / mars-project/mars

Integrate TensorFlow more seamlessly

Open
#796 0 comments 0 reactions 0 assignees View on GitHub
mod: learn type: feature
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

I think basically, we have two things to do to make integrating TensorFlow more seamlessly.

1. Support start TensorFlow cluster in Mars cluster. API like:

```
from mars.learn.contrib.tensorflow import start_tf_cluster

start_tf_cluster(n_worker=3, n_ps=2)
```

The training of TensorFlow can be deployed to the cluster created by Mars.

2. Support dataset from Mars. We could implement sort of `MarsDataSet` by leveraging TensorFlow Dataset API so that the data from Mars tensor or DataFrame could be feed into TensorFlow for training.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing how Mars creates and manages its cluster, then review the TensorFlow Dataset API and Mars tensor/DataFrame interfaces. Done means defining and implementing both the start_tf_cluster API and a MarsDataSet path for TensorFlow training, with tests for each integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.