aspect-build / aspect-build/rules_py

[Bug]: py_binary not usable in kubernetes?

Open
#934 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Starlark
Stars
145
Forks
97
Avg merge
1d 1h
Merged PRs (30d)
71

Description

### What happened?

Hi folks,

I'm building a py_binary with aspiect_rules_py ver 1.11.2, bazel ver 9.0.0
In the BUILD file, I've got the following straightforward setup:

```
py_binary(
name = "wrapper",
srcs = [
"wrapper.py",
],
deps = [
"//lib/python/(..)",
"//server/(...)",
"@pypi//click",
(...)
],
)

py_image_layer(
name = "main_app",
binary = ":wrapper",
)

oci_image(
name = "docker_image",
# We need bash and python for this.
base = "@(...)",
entrypoint = ["/server/(...)/wrapper
exposed_ports = [],
tars = [":main_app"],
)

oci_load(
name = "docker_load",
image = ":docker_image",
repo_tags = [
"(...):local",
],
tags = [
"artifact",
],
)
```

It compiles fine, when running it with docker, it starts just fine.
However, when trying to deploy the docker image into kubernetes, it fails with the following error:
```
Error: x Unable to run command:
|-> Unable to create base venv directory
`-> Read-only file system (os error 30)
```

Which is right, docker has a writable layer on the container, while the local Kubernetes cluster doesn't: it runs with an immutable image (security!).

When starting the binary with a writeable layer mounted for its .venv directory (simulated with local docker), the error changes:
```
$ docker run -ti --read-only -v "/tmp/tmp.zI78FnzdIr:/server/(...)/wrapper.runfiles/.wrapper.venv" --entrypoint=/server/(...)/wrapper (...):local
Error: x Unable to run command:
|-> Unable to remove venv_root directory
`-> Read-only file system (os error 30)
```

I wonder, is there a best known way to parameterize either py_binary; py_image_layer, or k8s to be able to run a python_binary create with aspect rules?

### Version

Development (host) and target OS/architectures:

Output of `bazel --version`: 9.0.0

Version of the Aspect rules, or other relevant rules from your
`WORKSPACE` or `MODULE.bazel` file: 1.10.0 (but no difference with 1.11.2)

Language(s) and/or frameworks involved:
python

### How to reproduce

Simulate read-onliness of Kubernetes with `--read-only` flag:

1) Check out https://github.com/ha1dfo/repro-py-proto/
```shell
$ bazel run //application:docker_load

$ docker run -ti --read-only apple:local
Error: x Unable to run command:
|-> Unable to create base venv directory
`-> Read-only file system (os error 30)
```

### Any other information?
N/A

Contributor guide

Open the contributing guide

Research direction

Start with the linked reproducer and its //application:docker_load target, then inspect the application BUILD file's py_binary, py_image_layer, and oci_image setup. Run the image with Docker's --read-only flag and trace the virtual-environment writes and cleanup. Done means identifying a supported configuration or documenting the limitation for read-only Kubernetes containers.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
build-system, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.