pip install of tensorflow-text disables tensorflow-gpu
Open
@broken is already working on this.
Since Jan 6, 2020.
- Dominant language
- C++
- Stars
- 1.3k
- Forks
- 379
- Avg merge
- 3h 30m
- Merged PRs (30d)
- 8
Description
It seems that a pip install of tensorflow-text>=2.0.0rc0 also installs tensorflow-2. If you previously had installed tensorflow-gpu the new one will disable GPU access.
Steps to reproduce:
- Build a new docker image with tf-gpu:
Dockerfile
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root
Build with docker build -t prueba .
Test correct GPU access:
$ docker run --runtime=nvidia --rm -it prueba:latest python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 18:55:03.225780: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 18:55:03.252825: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 18:55:03.253637: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43890e0 executing computations on platform Host. Devices:
2019-12-06 18:55:03.253666: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-12-06 18:55:03.256129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-06 18:55:03.361065: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.362691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4460890 executing computations on platform CUDA. Devices:
2019-12-06 18:55:03.362794: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
2019-12-06 18:55:03.363437: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.364914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:02:00.0
2019-12-06 18:55:03.365825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 18:55:03.370683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-06 18:55:03.373270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-06 18:55:03.373944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-06 18:55:03.376933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-06 18:55:03.378858: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-06 18:55:03.384248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-06 18:55:03.384400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.384853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.385166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-06 18:55:03.385213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 18:55:03.385816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-06 18:55:03.385834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-06 18:55:03.385842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-06 18:55:03.385961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.386314: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 18:55:03.386633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 3330 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:02:00.0, compute capability: 5.0)
True
- New dockerfile with tensorflow-text:
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root
RUN pip install tensorflow-text>=2.0.0rc0
Build and test... no GPU:
$ docker run --runtime=nvidia --rm -it prueba:latest python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 19:02:11.488695: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 19:02:11.512879: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 19:02:11.513972: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3726f30 executing computations on platform Host. Devices:
2019-12-06 19:02:11.514007: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
False
- Workaround: uninstall tensorflow after installing tensorflow-text only to user. Dockerfile:
FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
WORKDIR /root
RUN pip install --user tensorflow-text>=2.0.0rc0
RUN pip uninstall -y tensorflow
build and test, tensorflow-gpu works
$ docker build -t prueba -f Dockerfile .
Sending build context to Docker daemon 4.096kB
Step 1/4 : FROM tensorflow/tensorflow:latest-gpu-py3-jupyter
---> 88178d65d12c
Step 2/4 : WORKDIR /root
---> Using cache
---> 39616c78086e
Step 3/4 : RUN pip install --user tensorflow-text>=2.0.0rc0
---> Running in a08e37ee49da
WARNING: The scripts saved_model_cli, tensorboard, tf_upgrade_v2, tflite_convert, toco and toco_from_protos are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container a08e37ee49da
---> d5c415130f01
Step 4/4 : RUN pip uninstall -y tensorflow
---> Running in 23e3d7e9a6a8
Uninstalling tensorflow-2.0.0:
Successfully uninstalled tensorflow-2.0.0
Removing intermediate container 23e3d7e9a6a8
---> 7b91c3f6eeed
Successfully built 7b91c3f6eeed
Successfully tagged prueba:latest
$ docker run --runtime=nvidia --rm -it prueba:latest python -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
2019-12-06 19:03:36.662969: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-06 19:03:36.688813: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592000000 Hz
2019-12-06 19:03:36.689431: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4c00060 executing computations on platform Host. Devices:
2019-12-06 19:03:36.689461: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-12-06 19:03:36.691759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-06 19:03:36.733931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.734501: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4cd7810 executing computations on platform CUDA. Devices:
2019-12-06 19:03:36.734534: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 960M, Compute Capability 5.0
2019-12-06 19:03:36.734750: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.735122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:02:00.0
2019-12-06 19:03:36.735374: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 19:03:36.737036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-06 19:03:36.738244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-06 19:03:36.738574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-06 19:03:36.740186: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-06 19:03:36.741472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-06 19:03:36.745057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-06 19:03:36.745242: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.745721: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.746093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-06 19:03:36.746178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-06 19:03:36.746822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-06 19:03:36.746837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-06 19:03:36.746847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-06 19:03:36.747053: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.747601: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-06 19:03:36.748010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 3330 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960M, pci bus id: 0000:02:00.0, compute capability: 5.0)
True
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.