apache / apache/beam

[Feature Request]: Host docker images with the `conda` package manager for Beam's Python SDK.

Open
#22,349 26 comments 0 reactions 0 assignees View on GitHub
core new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Acquiring scientific dependencies in the Python ecosystem is challenging. `pip` and `apt-get` alone are not sufficient, for various reasons, the most significant of which is due to community. The scientific python community has standardized to one package manager: Anaconda. Within that package manager, most scientific software is built and distributed via conda-forge.

Given this, I propose the following: The Apache Beam Project should builds a new set of Docker images that include a `conda` manage python environment. The Dockerfile for the containers could look like [this](https://github.com/google/weather-tools/blob/main/weather_mv/Dockerfile):
```
ARG py_version=3.8
FROM apache/beam_python${py_version}_sdk:2.40.0 as beam_sdk
FROM continuumio/miniconda3:4.12.0
ARG py_version

# Update miniconda
RUN conda update conda -y

# Install desired python version
RUN conda install python=${py_version} -y

# Install SDK.
RUN pip install --no-cache-dir apache-beam[gcp]==2.40.0

# Verify that the image does not have conflicting dependencies.
RUN pip check

# Copy files from official SDK image, including script/dependencies.
COPY --from=beam_sdk /opt/apache/beam /opt/apache/beam

# Set the entrypoint to Apache Beam SDK launcher.
ENTRYPOINT ["/opt/apache/beam/boot"]
```

From such an image, Python SDK users will gain immense flexibility in adding dependencies to their Beam runtime environment (especially Dataflow, and likely including all remote Beam runners). For example, adding [a genuinely difficult-to-install dependency](https://github.com/ecmwf/metview-python/issues/19#issuecomment-874990120) would be as easy as adding

```
conda install -c conda-forge -y
```

to a `setup.py` file (following the [CUSTOM_COMMANDS pattern](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython)).

## Why should Apache Beam do this and not another third party?

This is a unique application of custom containers in Beam. Instead of an image with specific dependencies for an application, this package manager can obtain nearly all dependencies in the Python ecosystem. I argue that it makes sense for a member of the Apache project, or similarly open and community-federated project, manage and host this image in order to guard against potential supply chain attacks. Further, including `conda` as a Python SDK runtime environment would accelerate dependency management, especially of the PyData stack, on Apache Beam: It would help avoid the creation of lots of similar Docker images (to host each specific dependency, or else, to duplicate hosting `conda`).

### Issue Priority

Priority: 3

### Issue Component

Component: sdk-py-core

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.