bazelbuild / bazelbuild/bazel

Using Bazel to build jaxlib on Windows: cannot complete build through proxy server

Open
#15,740 5 comments 0 reactions 0 assignees View on GitHub
area-Windows P2 team-ExternalDeps type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

**System Info**
- Windows 10
- Python 3.9.12
- Bazel 5.1.1

I am building JAX from source on Windows but I cannot complete the build due to an SSL certificate issue. My environment variables are set up to work with my proxy, and I've configured various other programs to work with it as well. I need help configuring Bazel to work with my proxy.

When I attempt to build jaxlib with the command `python .\build\build.py`
I get the following output:
```
_ _ __ __
| | / \ \ \/ /
_ | |/ _ \ \ /
| |_| / ___ \/ \
\___/_/ \/_/\_\

Bazel binary path: C:\ProgramData\chocolatey\bin\bazel.EXE
Bazel version: 5.1.1
Python binary path: C:/Users/dmsol/Anaconda3/python.exe
Python version: 3.9
NumPy version: 1.21.5
MKL-DNN enabled: yes
Target CPU: AMD64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
C:\ProgramData\chocolatey\bin\bazel.EXE run --verbose_failures=true --config=mkl_open_source_only :build_wheel -- --output_path=C:\Users\dmsol\Documents\ROTE\jax\dist --cpu=AMD64
WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead
INFO: Reading 'startup' options from c:\users\dmsol\documents\rote\jax\.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=C:Program Files (x86)Javajre1.8.0_261libsecuritycacerts, --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit
INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=80
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Options provided by the client:
Inherited 'build' options: --python_path=C:/Users/dmsol/Anaconda3/python.exe
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.jax_configure.bazelrc:
Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=C:/Users/dmsol/Anaconda3/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/dmsol/Anaconda3/python.exe --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file c:\users\dmsol\documents\rote\jax\.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file c:\users\dmsol\documents\rote\jax\.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:windows in file c:\users\dmsol\documents\rote\jax\.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++17 --host_cxxopt=/std:c++17 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true
INFO: Repository org_tensorflow instantiated at:
C:/users/dmsol/documents/rote/jax/WORKSPACE:8:13: in
Repository rule http_archive defined at:
C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in
WARNING: Download from https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz failed: class javax.net.ssl.SSLHandshakeException PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: An error occurred during the fetch of repository 'org_tensorflow':
Traceback (most recent call last):
File "C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: C:/users/dmsol/documents/rote/jax/WORKSPACE:8:13: fetching http_archive rule //external:org_tensorflow: Traceback (most recent call last):
File "C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false
WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead
ERROR: @org_tensorflow//tensorflow/compiler/xla/python:enable_gpu :: Error loading option @org_tensorflow//tensorflow/compiler/xla/python:enable_gpu: no such package '@org_tensorflow//tensorflow': java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
b''
Traceback (most recent call last):
File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 528, in
main()
File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 523, in main
shell(command)
File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "C:\Users\dmsol\Anaconda3\lib\subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\dmsol\Anaconda3\lib\subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\ProgramData\\chocolatey\\bin\\bazel.EXE', 'run', '--verbose_failures=true', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=C:\\Users\\dmsol\\Documents\\ROTE\\jax\\dist', '--cpu=AMD64']' returned non-zero exit status 2.
```

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Set up a proxy server / firewall. Follow [these instructions](https://jax.readthedocs.io/en/latest/developer.html#additional-notes-for-building-jaxlib-from-source-on-windows) to build JAX from source on Windows for CPU only. For Bazel, I followed [these](https://bazel.build/install/bazelisk) instructions to install Bazelisk on my device.

After configuring the proxy for pip, MSYS2, git, and my device (using environment variables), I followed the JAX instructions, opened PowerShell as admin, and ran the command:
```
python .\build\build.py
```

### Which operating system are you running Bazel on?

Windows 10

### What is the output of `bazel info release`?

Starting local Bazel server and connecting to it... WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead INFO: Reading 'startup' options from c:\users\dmsol\documents\rote\jax\.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=C:Program Files (x86)Javajre1.8.0_261libsecuritycacerts, --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit INFO: Options provided by the client: Inherited 'common' options: --isatty=1 --terminal_columns=213 INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax\.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Options provided by the client: Inherited 'build' options: --python_path=C:/Users/dmsol/Anaconda3/python.exe INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax\.bazelrc: Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax\.jax_configure.bazelrc: Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=C:/Users/dmsol/Anaconda3/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/dmsol/Anaconda3/python.exe --distinct_host_configuration=false INFO: Found applicable config definition build:short_logs in file c:\users\dmsol\documents\rote\jax\.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:windows in file c:\users\dmsol\documents\rote\jax\.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++17 --host_cxxopt=/std:c++17 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true WARNING: info command does not support starlark options. Ignoring options: [--@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false, --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false] release 5.1.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
git remote get-url origin --> https://github.com/google/jax

git rev-parse main --> 1089c792d5e84403ee3eddcfeed15ec33dd28f20

git rev-parse HEAD --> 1089c792d5e84403ee3eddcfeed15ec33dd28f20
```

### Have you found anything relevant by searching the web?

**Similar Issues**
- [Unable to find valid certification path](https://github.com/tensorflow/tensorflow/issues/54899)
- [Bazel build behind proxy](https://stackoverflow.com/questions/52311486/bazel-build-behind-proxy)
- [Work behind a proxy](https://github.com/bazelbuild/bazel/issues/601)
- [tensorflow building issue on ubuntu 14.04: Accessing remote repository through a proxy](https://github.com/bazelbuild/bazel/issues/587)

### Any other information, logs, or outputs that you want to share?

I have also tried cloning the dependencies to my local device and pointing the repository variables by using:
```
python build/build.py --bazel_options=--override_repository=org_tensorflow=/path/to/tensorflow
```
and the like for the remaining dependencies. This gets rid of the SSL error but end of giving a slew of other errors. Besides, I really need this to specifically work through the proxy so that others at my company can replicate the process with relative ease.

Contributor guide

Open the contributing guide

Research direction

Start with build/build.py and WORKSPACE line 8, then inspect the .bazelrc JVM trustStore settings shown in the log. Reproduce with Python 3.9, Bazel 5.1.1, Windows 10, and a proxy by running python .\build\build.py; done means the TensorFlow archive fetch and jaxlib build complete without the PKIX certificate error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.