google-deepmind / google-deepmind/reverb
TensorFlow ROCm support?
- Dominant language
- C++
- Stars
- 793
- Forks
- 113
- PR merge metrics
- No merged PRs in 30d
Description
Hello everyone,
recently I tried reverb with the AMD GPU enhanced Tensorflow.
Doing it the streight forward way:
In completely new venv
- pip3 install tensorflow-rocm
- pip3 install dm-reverb
got me this error:
```
>>> import tensorflow
2023-01-21 14:18:31.516305: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> import reverb
Traceback (most recent call last):
File "/tmp/venv/lib/python3.10/site-packages/reverb/pybind.py", line 4, in
from .libpybind import *
ImportError: /tmp/venv/lib/python3.10/site-packages/reverb/libschema_cc_proto.so: undefined symbol: _ZNK6google8protobuf7Message25InitializationErrorStringB5cxx11Ev
```
So I tried to recompile reverb myself by basically using an AMD provided docker base image:
ARG cpu_base_image="rocm/tensorflow-build:latest-python3.10-rocm5.4.0".
With a little tinkering (I will provide the details if necessary) everything compiled. But now I'm stuck at linking with the message below.
It's a bit strange as the local compiler seems to have those symbols defined in its libraries.
```
ERROR: /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/external/com_github_grpc_grpc/src/compiler/BUILD:80:18: Linking of rule '@com_github_grpc_grpc//src/compiler:grpc_cpp_plugin' failed (Exit 1): process-wrapper failed: error executing command
(cd /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/sandbox/processwrapper-sandbox/548/execroot/reverb && \
exec env - \
LD_LIBRARY_PATH=/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/usr/local/lib64 \
PATH=/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/root/.cache/bazel/_bazel_root/install/46850c2a96e4b4b07623822a03209f74/process-wrapper '--timeout=0' '--kill_delay=15' /opt/rh/devtoolset-9/root/usr/bin/gcc @bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin-2.params) process-wrapper failed: error executing command
(cd /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/sandbox/processwrapper-sandbox/548/execroot/reverb && \
exec env - \
LD_LIBRARY_PATH=/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/usr/local/lib64 \
PATH=/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/root/.cache/bazel/_bazel_root/install/46850c2a96e4b4b07623822a03209f74/process-wrapper '--timeout=0' '--kill_delay=15' /opt/rh/devtoolset-9/root/usr/bin/gcc @bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin-2.params)
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function grpc_cpp_generator::ClassName(google::protobuf::Descriptor const*, bool): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function ProtoBufFile::package_parts() const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function CppGrpcGenerator::Generate(google::protobuf::FileDescriptor const*, std::string const&, google::protobuf::compiler::GeneratorContext*, std::string*) const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function CppGrpcGenerator::Generate(google::protobuf::FileDescriptor const*, std::string const&, google::protobuf::compiler::GeneratorContext*, std::string*) const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/dynamic_message.o:dynamic_message.cc:function google::protobuf::DynamicMessageFactory::GetPrototypeNoLock(google::protobuf::Descriptor const*) [clone .cold]: error: undefined reference to '__cxa_throw_bad_array_new_length'
collect2: error: ld returned 1 exit status
Target //reverb/pip_package:build_pip_package failed to build
```
Contributor guide
Research direction
Start with reverb/pybind.py and the pip installation commands, reproducing the import failure with tensorflow-rocm and dm-reverb. Then inspect the //reverb/pip_package:build_pip_package target and its Bazel dependencies using the reported ROCm Docker image. Done means determining whether ROCm TensorFlow is supported and identifying a reproducible resolution or documented limitation for both failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, tensorflow
- Domain
- build-system, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100