tensorflow / tensorflow/java

arm64 and x86_64 linux: TF java full native builds are failing to find the native headers

未关闭
#544 8 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Java
星标
928
派生
227
PR 合并指标
30 天内没有已合并 PR

描述

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04 x86_64): Linux Ubuntu 22.04, aarch64
  • TensorFlow installed from (source or binary): Source
  • TensorFlow version: TF version is v2.16.1, TF java version is tag: v1.0.0-rc.1
  • Java version (i.e., the output of java -version): openjdk 11.0.23 2024-04-16
  • Java command line flags (e.g., GC parameters): mvn install -P native-build -Dbazel.build.flags='--verbose_failures -s --config=mkl_aarch64_threadpool' -X
  • Installed from Maven Central?: No
  • Bazel version (if compiling from source): bazel 6.5.0
  • GCC/Compiler version (if compiling from source):gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
  • CUDA/cuDNN version: Not applicable
  • GPU model and memory: Not applicable

Describe the problem
TensorFlow java source builds are failing on aarch64 linux system with the missing native headers. please let me know how it's built for x86_64 linux platform.

based on my debugging so far it looks like the dependency comes from this commit which added C API extension for custom gradient functions, and introduced these headers and .cc which requires several third_party libraries from tensorflow native but none of those bazel workspaces are cloned.


tensorflow-core/tensorflow-core-native/src/main/native/org/tensorflow/internal/c_api$ 
tfj_gradients.h  tfj_gradients_impl.cc  tfj_graph.h  tfj_graph_impl.cc  tfj_scope.h  tfj_scope_impl.cc

I tried to manually clone the missing workspaces into bazel cache, but the cycle is never ending, it's missing tsl, eigen, ml_dtype, absl, protobuf, and now compiled headers for protobuf....

Provide the exact sequence of commands / steps that you executed before running into the problem

sudo apt-get install pkg-config ccache clang ant python3-pip swig git file wget unzip tar bzip2 gzip patch autoconf-archive autogen automake make cmake libtool bison flex perl nasm curl gfortran libasound2-dev freeglut3-dev libgtk2.0-dev libusb-dev zlib1g libffi-dev libbz2-dev zlib1g-dev

sudo apt install maven default-jdk

cd $HOME
mkdir bazel
cd bazel
wget https://github.com/bazelbuild/bazel/releases/download/6.5.0/bazel-6.5.0-linux-arm64
mv bazel-6.5.0-linux-arm64 bazel
chmod a+x bazel
export PATH=/home/ubuntu/bazel/:$PATH

# Build and install javacpp-presets.
# Clone the following forked repo to exclude the libraries that are not supported and not required
git clone https://github.com/snadampal/javacpp-presets.git
cd javacpp-presets
git checkout tfjava_aarch64
mvn install -Djavacpp.platform=linux-arm64 -Dmaven.javadoc.skip=true -X -T 16

# Build and install tensorflow java bindings
git clone https://github.com/tensorflow/java.git
cd java
git checkout v1.0.0-rc.1
mvn install -P native-build -Dbazel.build.flags='--verbose_failures -s --config=mkl_aarch64_threadpool' -X

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从报告中列出的 tensorflow-core 原生文件以及 v1.0.0-rc.1 原生构建的 Maven/Bazel 命令开始。在 Linux arm64 上重现缺少头文件的失败,然后将构建行为与报告中的 x86_64 情况进行比较。当原生 Java 构建解析其依赖项并成功完成时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
build-system
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。