aspect-build / aspect-build/rules_py
[Bug]: Running same binary in container build with `py_image_layer` causes failed imports
- Dominant language
- Starlark
- Stars
- 145
- Forks
- 97
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 71
Description
### What happened?
We recently migrated from version 0.7.1 to 1.2.1 and migrated the way we build our docker images from using a modified version of the old template to `py_image_layer`. Overall it has been great except for one thing: we deploy our containers to K8S and have health/status checks on them. The issue is the health checks use the same binary that the image runs normally, just in different modes. Concretely, we are running [Dagster](https://dagster.io/) which has the container run `dagster api grpc` and then the health checks use `dagster api grpc-health-check`. What we found is since the same binary target is being run by two separate cases in the same container, the venv that backed the python script was being re-created each health check run. This caused the base process to lose the packages in it's venv temporarily, causing it to be unhealthy and thus fail.
It seems like #522 would fix this since the venv would be stable but in the meantime, is there a way to fix this temporarily?
### Version
Development (host) and target OS/architectures: aarch Darwin -> aarch Darwin, Linux x86_64 -> Linux x86_64
Output of `bazel --version`: 8.0.0
Version of the Aspect rules, or other relevant rules from your
`WORKSPACE` or `MODULE.bazel` file: 1.2.1
Language(s) and/or frameworks involved: Python 3.11, Docker/rules_oci 1.7.4
### How to reproduce
```shell
Hard to reliably reproduce since it is a bit of a race condition.
One way would be to have a python binary that continually tries to import a package and create an OCI image using `py_image_layer`. Then run the image and then `exec` the binary again in another window. One of the two should error out but may take several iterations
```
### Any other information?
We were getting errors that looked like
```
ERROR 2025-02-05T17:31:44.082500645Z [resource.labels.containerName: dagster-user-deployments] File "/data/pomelo/dagster.runfiles/.dagster.venv/lib/python3.11/site-packages/dsp/modules/__init__.py", line 22, in
ERROR 2025-02-05T17:31:44.082966540Z [resource.labels.containerName: dagster-user-deployments] from .pyserini import *
ERROR 2025-02-05T17:31:44.082988358Z [resource.labels.containerName: dagster-user-deployments] File "/data/pomelo/dagster.runfiles/.dagster.venv/lib/python3.11/site-packages/dsp/modules/pyserini.py", line 4, in
ERROR 2025-02-05T17:31:44.083429066Z [resource.labels.containerName: dagster-user-deployments] from datasets import Dataset
ERROR 2025-02-05T17:31:44.083460Z [resource.labels.containerName: dagster-user-deployments] ModuleNotFoundError: No module named 'datasets'
```
despite having `datasets` included in the binary. After turning off our health checks, the error went away. I also SSH'd into the pod and inspected the packages in the venv that was generated and saw it would repeatedly have a subset of the expected packages and then quickly after have all of the expected packages
Contributor guide
Assessment
This issue has not been assessed yet.