tensorflow / tensorflow/probability

Distributed Training in TensorFlow Probability

Open
#496 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

tensorflow 2.0
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
1.1k
No merged PRs in 30d

Description

Hi everyone.

This is my first attempt fitting a model using Tensorflow Probability with distributed strategy.

This issue happens on python=3.6.8; tensorflow=2.0.0-beta1; tensorflow_probablity=0.7.0, and also on python=3.6.8; tensorflow=1.14.1-dev20190614; tensorflow_probablity=0.8.0-dev20190614.

We have the following GPU topology nvidia-smi topo -m, and all the GPUs are GeForce RTX 2080 Ti:

	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity
GPU0	 X 	PIX	NODE	NODE	SYS	SYS	SYS	SYS	0-11,24-35
GPU1	PIX	 X 	NODE	NODE	SYS	SYS	SYS	SYS	0-11,24-35
GPU2	NODE	NODE	 X 	PIX	SYS	SYS	SYS	SYS	0-11,24-35
GPU3	NODE	NODE	PIX	 X 	SYS	SYS	SYS	SYS	0-11,24-35
GPU4	SYS	SYS	SYS	SYS	 X 	PIX	NODE	NODE	12-23,36-47
GPU5	SYS	SYS	SYS	SYS	PIX	 X 	NODE	NODE	12-23,36-47
GPU6	SYS	SYS	SYS	SYS	NODE	NODE	 X 	PIX	12-23,36-47
GPU7	SYS	SYS	SYS	SYS	NODE	NODE	PIX	 X 	12-23,36-47

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks

The model is a simple linear regression using Edward2 but, at this point

tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0

the process hangs up. I see a 0% volatile gpu-util but High GPU Memory-Usage nvidia-smi:


+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.14       Driver Version: 430.14       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:1A:00.0 Off |                  N/A |
| 27%   31C    P8    10W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:1B:00.0 Off |                  N/A |
| 27%   29C    P8    20W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:60:00.0 Off |                  N/A |
| 27%   29C    P8    21W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:61:00.0 Off |                  N/A |
| 27%   32C    P8    17W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 208...  Off  | 00000000:B1:00.0 Off |                  N/A |
| 27%   30C    P8     1W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 208...  Off  | 00000000:B2:00.0 Off |                  N/A |
| 27%   31C    P8     7W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce RTX 208...  Off  | 00000000:DA:00.0 Off |                  N/A |
| 27%   27C    P8     3W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce RTX 208...  Off  | 00000000:DB:00.0 Off |                  N/A |
| 27%   29C    P8     6W / 250W |  10663MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    171693      C   python                                     10653MiB |
|    1    171693      C   python                                     10653MiB |
|    2    171693      C   python                                     10653MiB |
|    3    171693      C   python                                     10653MiB |
|    4    171693      C   python                                     10653MiB |
|    5    171693      C   python                                     10653MiB |
|    6    171693      C   python                                     10653MiB |
|    7    171693      C   python                                     10653MiB |
+-----------------------------------------------------------------------------+

and top command shows

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
171693 abm       20   0  147,2g   8,7g 911268 S   0,3  4,7   0:36.67 python

Bellow, the model script and the full terminal output.

Any clue will be welcome! I've also looked for a simple model using TFP and distributed training without success :-(

The model script:


from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
# import pandas as pd
# import re
# from string import ascii_uppercase

import tensorflow as tf
from tensorflow_probability import edward2 as ed
# ed.set_seed(42)
N = 4000  # number of data points
D = 10  # number of features
w_true = np.random.randn(D)

regressors = np.random.randn(N, D).astype('float32')


def build_toy_dataset(x, w, noise_std=0.25):
    features = np.dot(x, w) + np.random.normal(
        0, noise_std, size=x.shape[0])
    labels = np.repeat(1, N)
    return features.astype(dtype='int32'), labels.astype(dtype='int32')


features_, labels_ = build_toy_dataset(regressors, w_true)

strategy = tf.distribute.MirroredStrategy()

print('Number of devices: {}'.format(strategy.num_replicas_in_sync))

session_config = tf.compat.v1.ConfigProto()
session_config.gpu_options.allow_growth = True
config = tf.estimator.RunConfig(
    train_distribute=strategy, eval_distribute=strategy,
    session_config=session_config)


def train_input_fn():
    """An input function for training"""
    # Convert the inputs to a Dataset.
    features = tf.data.Dataset.from_tensors(features_)
    labels = tf.data.Dataset.from_tensors(labels_)
    dataset = tf.data.Dataset.zip((features, labels))
    # Shuffle, repeat, and batch the examples.
    return dataset


def eval_input_fn():
    features = tf.data.Dataset.from_tensors(features_)
    labels = tf.data.Dataset.from_tensors(labels_)
    return tf.data.Dataset.zip((features, labels))


def predict_input_fn():
    predict_features = tf.data.Dataset.from_tensors(features_[0:10]).repeat(10)
    return predict_features


# def test_input_fn():
#     features = tf.data.Dataset.from_tensors(X_test)
#     outputs = tf.data.Dataset.from_tensors(Y_test)
#     dataset = tf.data.Dataset.zip((features, outputs))
#     dataset = dataset.shuffle(buffer_size=N)
#     return dataset


def build_model_fn_optimizer():
    """Simple model_fn with optimizer."""

    # TODO(anjalisridhar): Move this inside the model_fn once
    # OptimizerV2 is done?
    optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=0.05)

    def model_fn(features, labels, mode, params):

        regressors = params['regressors']

        def regression_model(input):
            w = ed.Normal(
                loc=tf.zeros(input.shape[1]),
                scale=tf.ones(input.shape[1]),
                name="w")

            b = ed.Normal(loc=0., scale=1., name="b")

            y = ed.Normal(
                loc=tf.tensordot(input, w, [[1], [0]]) + b,
                scale=tf.ones(N), name="y")
            return y

        log_joint = ed.make_log_joint_fn(regression_model)

        y = regression_model(regressors)

        if mode == tf.estimator.ModeKeys.PREDICT:
            predictions = {"y": y}
            return tf.estimator.EstimatorSpec(mode,
                                              predictions=predictions)

        var_1 = tf.Variable(name='var_1',
                            initial_value=regressors.shape[1]*[1.])
        var_2 = tf.Variable(name='var_2', initial_value=0.5)

        def loss_fn():
            return -log_joint(regressors, w=var_1, b=var_2, y=features)

        if mode == tf.estimator.ModeKeys.EVAL:
            return tf.estimator.EstimatorSpec(mode, loss=loss_fn())

        assert mode == tf.estimator.ModeKeys.TRAIN

        global_step = tf.compat.v1.train.get_global_step()
        train_op = optimizer.minimize(loss_fn(), global_step=global_step)
        return tf.estimator.EstimatorSpec(mode,
                                          loss=loss_fn(), train_op=train_op)

    return model_fn


steps = 2000

estimator = tf.estimator.Estimator(
    model_fn=build_model_fn_optimizer(),
    params={'regressors': regressors},
    config=config)

estimator.train(input_fn=train_input_fn, steps=steps)


eval_result = estimator.evaluate(input_fn=eval_input_fn, steps=steps)
print("Eval result: {}".format(eval_result))

The terminal output:


2019-07-15 11:02:29.217577: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-15 11:02:29.711779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
2019-07-15 11:02:29.713201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
2019-07-15 11:02:29.714576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:60:00.0
2019-07-15 11:02:29.715939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:61:00.0
2019-07-15 11:02:29.717302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 4 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b1:00.0
2019-07-15 11:02:29.718657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 5 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b2:00.0
2019-07-15 11:02:29.720020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 6 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:da:00.0
2019-07-15 11:02:29.721387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 7 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:db:00.0
2019-07-15 11:02:29.721667: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-15 11:02:29.723681: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-15 11:02:29.725517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-15 11:02:29.725857: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-15 11:02:29.728184: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-15 11:02:29.729922: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-15 11:02:29.735155: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-15 11:02:29.756397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7
2019-07-15 11:02:29.757044: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-07-15 11:02:32.001049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56435dd2ec20 executing computations on platform CUDA. Devices:
2019-07-15 11:02:32.001145: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001176: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001201: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (2): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001225: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (3): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001249: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (4): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001273: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (5): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001296: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (6): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.001320: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (7): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-07-15 11:02:32.017579: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2019-07-15 11:02:32.029129: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56435e6dcee0 executing computations on platform Host. Devices:
2019-07-15 11:02:32.029196: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-15 11:02:32.034236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
2019-07-15 11:02:32.035802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
2019-07-15 11:02:32.037373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:60:00.0
2019-07-15 11:02:32.038908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:61:00.0
2019-07-15 11:02:32.040448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 4 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b1:00.0
2019-07-15 11:02:32.041983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 5 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b2:00.0
2019-07-15 11:02:32.043515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 6 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:da:00.0
2019-07-15 11:02:32.044864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 7 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:db:00.0
2019-07-15 11:02:32.044925: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-15 11:02:32.044943: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-15 11:02:32.044960: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-15 11:02:32.044974: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-15 11:02:32.044991: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-15 11:02:32.045005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-15 11:02:32.045020: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-15 11:02:32.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7
2019-07-15 11:02:32.065247: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-15 11:02:32.078026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-15 11:02:32.078055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 4 5 6 7 
2019-07-15 11:02:32.078072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N N N N N 
2019-07-15 11:02:32.078083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N N N N N 
2019-07-15 11:02:32.078092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N N N N N 
2019-07-15 11:02:32.078103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N N N N N 
2019-07-15 11:02:32.078114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 4:   N N N N N N N N 
2019-07-15 11:02:32.078126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 5:   N N N N N N N N 
2019-07-15 11:02:32.078135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 6:   N N N N N N N N 
2019-07-15 11:02:32.078149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 7:   N N N N N N N N 
2019-07-15 11:02:32.089643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10310 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5)
2019-07-15 11:02:32.091397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10310 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:1b:00.0, compute capability: 7.5)
2019-07-15 11:02:32.093107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10310 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:60:00.0, compute capability: 7.5)
2019-07-15 11:02:32.095293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10310 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:61:00.0, compute capability: 7.5)
2019-07-15 11:02:32.097070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 10310 MB memory) -> physical GPU (device: 4, name: GeForce RTX 2080 Ti, pci bus id: 0000:b1:00.0, compute capability: 7.5)
2019-07-15 11:02:32.098801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 10310 MB memory) -> physical GPU (device: 5, name: GeForce RTX 2080 Ti, pci bus id: 0000:b2:00.0, compute capability: 7.5)
2019-07-15 11:02:32.100877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 10310 MB memory) -> physical GPU (device: 6, name: GeForce RTX 2080 Ti, pci bus id: 0000:da:00.0, compute capability: 7.5)
2019-07-15 11:02:32.102587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 10310 MB memory) -> physical GPU (device: 7, name: GeForce RTX 2080 Ti, pci bus id: 0000:db:00.0, compute capability: 7.5)
2019-07-15 11:02:32.114294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
2019-07-15 11:02:32.115581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
2019-07-15 11:02:32.116866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:60:00.0
2019-07-15 11:02:32.118135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:61:00.0
2019-07-15 11:02:32.119390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 4 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b1:00.0
2019-07-15 11:02:32.120641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 5 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b2:00.0
2019-07-15 11:02:32.121900: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 6 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:da:00.0
2019-07-15 11:02:32.123183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 7 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:db:00.0
2019-07-15 11:02:32.123224: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-15 11:02:32.123239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-15 11:02:32.123253: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-15 11:02:32.123268: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-15 11:02:32.123282: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-15 11:02:32.123300: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-15 11:02:32.123314: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-15 11:02:32.143056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7
2019-07-15 11:02:32.145096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-15 11:02:32.145117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 4 5 6 7 
2019-07-15 11:02:32.145128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N N N N N 
2019-07-15 11:02:32.145138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N N N N N 
2019-07-15 11:02:32.145148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N N N N N 
2019-07-15 11:02:32.145158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N N N N N 
2019-07-15 11:02:32.145168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 4:   N N N N N N N N 
2019-07-15 11:02:32.145180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 5:   N N N N N N N N 
2019-07-15 11:02:32.145189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 6:   N N N N N N N N 
2019-07-15 11:02:32.145203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 7:   N N N N N N N N 
2019-07-15 11:02:32.156583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 10310 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5)
2019-07-15 11:02:32.157941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 10310 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:1b:00.0, compute capability: 7.5)
2019-07-15 11:02:32.159223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:2 with 10310 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:60:00.0, compute capability: 7.5)
2019-07-15 11:02:32.160499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:3 with 10310 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:61:00.0, compute capability: 7.5)
2019-07-15 11:02:32.161771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:4 with 10310 MB memory) -> physical GPU (device: 4, name: GeForce RTX 2080 Ti, pci bus id: 0000:b1:00.0, compute capability: 7.5)
2019-07-15 11:02:32.163045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:5 with 10310 MB memory) -> physical GPU (device: 5, name: GeForce RTX 2080 Ti, pci bus id: 0000:b2:00.0, compute capability: 7.5)
2019-07-15 11:02:32.164315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:6 with 10310 MB memory) -> physical GPU (device: 6, name: GeForce RTX 2080 Ti, pci bus id: 0000:da:00.0, compute capability: 7.5)
2019-07-15 11:02:32.165582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:7 with 10310 MB memory) -> physical GPU (device: 7, name: GeForce RTX 2080 Ti, pci bus id: 0000:db:00.0, compute capability: 7.5)
Number of devices: 8
WARNING: Logging before flag parsing goes to stderr.
W0715 11:02:32.170489 140270170486592 estimator.py:1811] Using temporary folder as model directory: /tmp/tmpih_kqqs7
W0715 11:02:34.998128 140248969361152 deprecation.py:323] From /opt/anaconda3/envs/tf2_beta/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py:1340: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2019-07-15 11:02:35.817646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
2019-07-15 11:02:35.819076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
2019-07-15 11:02:35.820435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:60:00.0
2019-07-15 11:02:35.821815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:61:00.0
2019-07-15 11:02:35.823112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 4 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b1:00.0
2019-07-15 11:02:35.824403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 5 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b2:00.0
2019-07-15 11:02:35.825686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 6 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:da:00.0
2019-07-15 11:02:35.827024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 7 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:db:00.0
2019-07-15 11:02:35.827095: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-15 11:02:35.827120: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-15 11:02:35.827137: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-15 11:02:35.827154: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-15 11:02:35.827167: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-15 11:02:35.827186: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-15 11:02:35.827204: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-15 11:02:35.863675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7
2019-07-15 11:02:35.866846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-15 11:02:35.866876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1 2 3 4 5 6 7 
2019-07-15 11:02:35.866894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N N N N N N N N 
2019-07-15 11:02:35.866909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   N N N N N N N N 
2019-07-15 11:02:35.866924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2:   N N N N N N N N 
2019-07-15 11:02:35.866939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3:   N N N N N N N N 
2019-07-15 11:02:35.866985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 4:   N N N N N N N N 
2019-07-15 11:02:35.867000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 5:   N N N N N N N N 
2019-07-15 11:02:35.867014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 6:   N N N N N N N N 
2019-07-15 11:02:35.867048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 7:   N N N N N N N N 
2019-07-15 11:02:35.880930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10310 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5)
2019-07-15 11:02:35.882349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10310 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:1b:00.0, compute capability: 7.5)
2019-07-15 11:02:35.883761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10310 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:60:00.0, compute capability: 7.5)
2019-07-15 11:02:35.885120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10310 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:61:00.0, compute capability: 7.5)
2019-07-15 11:02:35.886546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 10310 MB memory) -> physical GPU (device: 4, name: GeForce RTX 2080 Ti, pci bus id: 0000:b1:00.0, compute capability: 7.5)
2019-07-15 11:02:35.887928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 10310 MB memory) -> physical GPU (device: 5, name: GeForce RTX 2080 Ti, pci bus id: 0000:b2:00.0, compute capability: 7.5)
2019-07-15 11:02:35.889355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 10310 MB memory) -> physical GPU (device: 6, name: GeForce RTX 2080 Ti, pci bus id: 0000:da:00.0, compute capability: 7.5)
2019-07-15 11:02:35.890771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 10310 MB memory) -> physical GPU (device: 7, name: GeForce RTX 2080 Ti, pci bus id: 0000:db:00.0, compute capability: 7.5)
2019-07-15 11:02:42.997038: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1483] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2019-07-15 11:02:46.463725: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied model script, especially tf.distribute.MirroredStrategy, train_input_fn, and estimator.train, using the listed Python, TensorFlow, and TensorFlow Probability versions. Reproduce the hang and compare the behavior around the libcublas load and GPU allocation. Done means distributed training completes and estimator.evaluate returns instead of stalling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.