tensorflow / tensorflow/cloud

Enable running tfc.run() on notebook running from within a AI Platform hosted notebook.

Open
#249 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
383
Forks
93
Avg merge
1d 3h
Merged PRs (30d)
1

Description

Using AI Platform hosted notebooks, we created an Jupyter notebook with the model that we are were planning to train and saved it. We created a separate notebook in which we had our runner wrapping script similar to

import tensorflow_cloud as tfc

tfc.run(
    docker_config=tfc.DockerConfig(
        image_build_bucket="somebucket",
        parent_image="gcr.io/xyz"), 
    entry_point="model.ipynb",
    distribution_strategy="auto",
    worker_count=5,
    requirements_txt='requirements.txt',
    chief_config=tfc.COMMON_MACHINE_CONFIGS["CPU"],
    worker_config=tfc.COMMON_MACHINE_CONFIGS["CPU"],
    job_labels={
        "job": "kaggle_competition",
        "team": "base_line",
    },
    stream_logs=False
)

The run fails with error

/opt/conda/lib/python3.7/site-packages/tensorflow_cloud/core/preprocess.py in _get_colab_notebook_content()
    207 def _get_colab_notebook_content():
    208     """Returns the colab notebook python code contents."""
--> 209     response = _message.blocking_request("get_ipynb",
    210                                          request="",
    211                                          timeout_sec=200)

AttributeError: 'NoneType' object has no attribute 'blocking_request'

Would be nice to add support for this case were all requirements and a proper base image are directly provided for the remote run.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorflow_cloud/core/preprocess.py at _get_colab_notebook_content() and trace how tfc.run() obtains notebook content. Reproduce the failure from an AI Platform hosted Jupyter notebook using the shown DockerConfig and entry_point settings. Done means the hosted-notebook case can run with the supplied requirements file and base image without relying on the unavailable Colab message interface.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.