aspect-build / aspect-build/rules_py
[Bug]: py_binary not usable in kubernetes?
- Dominant language
- Starlark
- Stars
- 145
- Forks
- 97
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 71
Description
### What happened?
Hi folks,
I'm building a py_binary with aspiect_rules_py ver 1.11.2, bazel ver 9.0.0
In the BUILD file, I've got the following straightforward setup:
```
py_binary(
name = "wrapper",
srcs = [
"wrapper.py",
],
deps = [
"//lib/python/(..)",
"//server/(...)",
"@pypi//click",
(...)
],
)
py_image_layer(
name = "main_app",
binary = ":wrapper",
)
oci_image(
name = "docker_image",
# We need bash and python for this.
base = "@(...)",
entrypoint = ["/server/(...)/wrapper
exposed_ports = [],
tars = [":main_app"],
)
oci_load(
name = "docker_load",
image = ":docker_image",
repo_tags = [
"(...):local",
],
tags = [
"artifact",
],
)
```
It compiles fine, when running it with docker, it starts just fine.
However, when trying to deploy the docker image into kubernetes, it fails with the following error:
```
Error: x Unable to run command:
|-> Unable to create base venv directory
`-> Read-only file system (os error 30)
```
Which is right, docker has a writable layer on the container, while the local Kubernetes cluster doesn't: it runs with an immutable image (security!).
When starting the binary with a writeable layer mounted for its .venv directory (simulated with local docker), the error changes:
```
$ docker run -ti --read-only -v "/tmp/tmp.zI78FnzdIr:/server/(...)/wrapper.runfiles/.wrapper.venv" --entrypoint=/server/(...)/wrapper (...):local
Error: x Unable to run command:
|-> Unable to remove venv_root directory
`-> Read-only file system (os error 30)
```
I wonder, is there a best known way to parameterize either py_binary; py_image_layer, or k8s to be able to run a python_binary create with aspect rules?
### Version
Development (host) and target OS/architectures:
Output of `bazel --version`: 9.0.0
Version of the Aspect rules, or other relevant rules from your
`WORKSPACE` or `MODULE.bazel` file: 1.10.0 (but no difference with 1.11.2)
Language(s) and/or frameworks involved:
python
### How to reproduce
Simulate read-onliness of Kubernetes with `--read-only` flag:
1) Check out https://github.com/ha1dfo/repro-py-proto/
```shell
$ bazel run //application:docker_load
$ docker run -ti --read-only apple:local
Error: x Unable to run command:
|-> Unable to create base venv directory
`-> Read-only file system (os error 30)
```
### Any other information?
N/A
Contributor guide
Research direction
Start with the linked reproducer and its //application:docker_load target, then inspect the application BUILD file's py_binary, py_image_layer, and oci_image setup. Run the image with Docker's --read-only flag and trace the virtual-environment writes and cleanup. Done means identifying a supported configuration or documenting the limitation for read-only Kubernetes containers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, python
- Domain
- build-system, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100