tensorflow / tensorflow/cloud

tfc.remote() == False if python file entrypoint and distribution_strategy is None

Open
#391 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
383
Forks
93
Avg merge
1d 3h
Merged PRs (30d)
1

Description

The 'get_preprocessed_entry_point" isn't run if the distribution strategy is None and the entry point is a python file (ends in ".py"):

https://github.com/tensorflow/cloud/blob/master/src/python/tensorflow_cloud/core/run.py#L266-L282

This is a problem because this is the place where the TF_KERAS_RUNNING_REMOTELY is injected to the entrypoint. Because this isn't set, calls to tfc.remote() in the user's script won't work in the way expected (i.e., will always return False).

Proposed Solution:
Inject this into the Dockerfile directly instead as an ENV var and pass in when building the image (in ContainerBuilder._create_docker_file).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/python/tensorflow_cloud/core/run.py around lines 266-282 to trace the Python-file entrypoint path when distribution_strategy is None. Then inspect ContainerBuilder._create_docker_file and verify that the TF_KERAS_RUNNING_REMOTELY environment variable reaches the entrypoint and makes tfc.remote() behave as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, tensorflow
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.