tensorflow / tensorflow/text

pip install of tensorflow-text disables tensorflow-gpu

Open
#195 5 comments 3 reactions 1 assignee View on GitHub

@broken is already working on this.

Since Jan 6, 2020.

Dominant language
C++
Stars
1.3k
Forks
379
Avg merge
3h 30m
Merged PRs (30d)
8

Description

It seems that a pip install of tensorflow-text>=2.0.0rc0 also installs tensorflow-2. If you previously had installed tensorflow-gpu the new one will disable GPU access.

Steps to reproduce:

  1. Build a new docker image with tf-gpu:
    Dockerfile
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root

Build with docker build -t prueba .

Test correct GPU access:

$ docker run --runtime=nvidia --rm -it prueba:latest  python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 18:55:03.225780: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 18:55:03.252825: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 18:55:03.253637: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43890e0 executing computations on platform Host. Devices:
2019-12-06 18:55:03.253666: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-12-06 18:55:03.256129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-06 18:55:03.361065: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.362691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4460890 executing computations on platform CUDA. Devices:
2019-12-06 18:55:03.362794: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
2019-12-06 18:55:03.363437: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.364914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:02:00.0
2019-12-06 18:55:03.365825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 18:55:03.370683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-06 18:55:03.373270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-06 18:55:03.373944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-06 18:55:03.376933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-06 18:55:03.378858: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-06 18:55:03.384248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-06 18:55:03.384400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.384853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.385166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-06 18:55:03.385213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 18:55:03.385816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-06 18:55:03.385834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-12-06 18:55:03.385842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-12-06 18:55:03.385961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.386314: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.386633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 3330 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:02:00.0, compute capability: 5.0)
True
  1. New dockerfile with tensorflow-text:
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root

RUN pip install tensorflow-text>=2.0.0rc0

Build and test... no GPU:

$ docker run --runtime=nvidia --rm -it prueba:latest  python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 19:02:11.488695: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 19:02:11.512879: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 19:02:11.513972: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3726f30 executing computations on platform Host. Devices:
2019-12-06 19:02:11.514007: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
False
  1. Workaround: uninstall tensorflow after installing tensorflow-text only to user. Dockerfile:
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root

RUN pip install --user tensorflow-text>=2.0.0rc0
RUN pip uninstall -y tensorflow

build and test, tensorflow-gpu works

$ docker build -t prueba -f Dockerfile .
Sending build context to Docker daemon  4.096kB
Step 1/4 : FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
 ---> 88178d65d12c
Step 2/4 : WORKDIR /root
 ---> Using cache
 ---> 39616c78086e
Step 3/4 : RUN pip install --user tensorflow-text>=2.0.0rc0
 ---> Running in a08e37ee49da
  WARNING: The scripts saved_model_cli, tensorboard, tf_upgrade_v2, tflite_convert, toco and toco_from_protos are installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container a08e37ee49da
 ---> d5c415130f01
Step 4/4 : RUN pip uninstall -y tensorflow
 ---> Running in 23e3d7e9a6a8
Uninstalling tensorflow-2.0.0:
  Successfully uninstalled tensorflow-2.0.0
Removing intermediate container 23e3d7e9a6a8
 ---> 7b91c3f6eeed
Successfully built 7b91c3f6eeed
Successfully tagged prueba:latest
$ docker run --runtime=nvidia --rm -it prueba:latest  python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 19:03:36.662969: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 19:03:36.688813: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 19:03:36.689431: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4c00060 executing computations on platform Host. Devices:
2019-12-06 19:03:36.689461: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-12-06 19:03:36.691759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-06 19:03:36.733931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.734501: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4cd7810 executing computations on platform CUDA. Devices:
2019-12-06 19:03:36.734534: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
2019-12-06 19:03:36.734750: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.735122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:02:00.0
2019-12-06 19:03:36.735374: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 19:03:36.737036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-06 19:03:36.738244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-06 19:03:36.738574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-06 19:03:36.740186: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-06 19:03:36.741472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-06 19:03:36.745057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-06 19:03:36.745242: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.745721: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.746093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-06 19:03:36.746178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 19:03:36.746822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-06 19:03:36.746837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-12-06 19:03:36.746847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-12-06 19:03:36.747053: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.747601: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.748010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 3330 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:02:00.0, compute capability: 5.0)
True

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.