google-deepmind / google-deepmind/reverb

TensorFlow ROCm support?

Open
#120 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
793
Forks
113
PR merge metrics
No merged PRs in 30d

Description

Hello everyone,

recently I tried reverb with the AMD GPU enhanced Tensorflow.
Doing it the streight forward way:
In completely new venv
- pip3 install tensorflow-rocm
- pip3 install dm-reverb

got me this error:
```
>>> import tensorflow
2023-01-21 14:18:31.516305: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> import reverb
Traceback (most recent call last):
File "/tmp/venv/lib/python3.10/site-packages/reverb/pybind.py", line 4, in
from .libpybind import *
ImportError: /tmp/venv/lib/python3.10/site-packages/reverb/libschema_cc_proto.so: undefined symbol: _ZNK6google8protobuf7Message25InitializationErrorStringB5cxx11Ev
```

So I tried to recompile reverb myself by basically using an AMD provided docker base image:
ARG cpu_base_image="rocm/tensorflow-build:latest-python3.10-rocm5.4.0".

With a little tinkering (I will provide the details if necessary) everything compiled. But now I'm stuck at linking with the message below.
It's a bit strange as the local compiler seems to have those symbols defined in its libraries.

```
ERROR: /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/external/com_github_grpc_grpc/src/compiler/BUILD:80:18: Linking of rule '@com_github_grpc_grpc//src/compiler:grpc_cpp_plugin' failed (Exit 1): process-wrapper failed: error executing command
(cd /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/sandbox/processwrapper-sandbox/548/execroot/reverb && \
exec env - \
LD_LIBRARY_PATH=/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/usr/local/lib64 \
PATH=/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/root/.cache/bazel/_bazel_root/install/46850c2a96e4b4b07623822a03209f74/process-wrapper '--timeout=0' '--kill_delay=15' /opt/rh/devtoolset-9/root/usr/bin/gcc @bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin-2.params) process-wrapper failed: error executing command
(cd /root/.cache/bazel/_bazel_root/a8a0a4aa310fa7eb496b741cf02da395/sandbox/processwrapper-sandbox/548/execroot/reverb && \
exec env - \
LD_LIBRARY_PATH=/opt/rh/devtoolset-9/root/usr/lib64:/opt/rh/devtoolset-9/root/usr/lib:/opt/rh/devtoolset-9/root/usr/lib64/dyninst:/opt/rh/devtoolset-9/root/usr/lib/dyninst:/usr/local/lib64 \
PATH=/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/root/.cache/bazel/_bazel_root/install/46850c2a96e4b4b07623822a03209f74/process-wrapper '--timeout=0' '--kill_delay=15' /opt/rh/devtoolset-9/root/usr/bin/gcc @bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin-2.params)
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function grpc_cpp_generator::ClassName(google::protobuf::Descriptor const*, bool): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function ProtoBufFile::package_parts() const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function CppGrpcGenerator::Generate(google::protobuf::FileDescriptor const*, std::string const&, google::protobuf::compiler::GeneratorContext*, std::string*) const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/_objs/grpc_cpp_plugin/cpp_plugin.o:cpp_plugin.cc:function CppGrpcGenerator::Generate(google::protobuf::FileDescriptor const*, std::string const&, google::protobuf::compiler::GeneratorContext*, std::string*) const: error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/dynamic_message.o:dynamic_message.cc:function google::protobuf::DynamicMessageFactory::GetPrototypeNoLock(google::protobuf::Descriptor const*) [clone .cold]: error: undefined reference to '__cxa_throw_bad_array_new_length'
collect2: error: ld returned 1 exit status
Target //reverb/pip_package:build_pip_package failed to build
```

Contributor guide

Open the contributing guide

Research direction

Start with reverb/pybind.py and the pip installation commands, reproducing the import failure with tensorflow-rocm and dm-reverb. Then inspect the //reverb/pip_package:build_pip_package target and its Bazel dependencies using the reported ROCm Docker image. Done means determining whether ROCm TensorFlow is supported and identifying a reproducible resolution or documented limitation for both failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, tensorflow
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.