apache / apache/airflow

Replace docker-compose breeze setup with testcontainers

Open
#43,514 0 comments 1 reaction 0 assignees View on GitHub
area:dev-env area:dev-tools
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

Part of `breeze` is a whole set of docker-compose files that are used to setup environment where various databases and integrations are set-up. This feature uses composability (sic!) of docker compose files and allows us to start breeze with the right set of database, integrations as docker-compose containers running together.

However there is a new tool called testcontainers https://testcontainers.com/?language=python that seem to have similar aim, but rather than using declarative and composable docker compose yaml files, they utiise native programming language integration to launch and shut-down necessary containers during running tests.

We could employ it to replace what breeze currently does for that part of our dev environment, and the added benefit is that you would not have to use breeze if you would like to run tests with "postgres", "kafka" etc. You would be able to do it easily using regular unit tests.

The part of breeze where it installs and manage the common container with all dependencies would still remain, but we could - I think fairly easily simplify and replace the part of Breeze where necessary containers are started via `--backend` and `--integration` database and plug them in pytest fixtures that could start/teardown necessary containers as needed even outside of breeze.

Contributor guide

Open the contributing guide

Research direction

Start by reading the docker-compose files and the Breeze paths handling the --backend and --integration options. Trace how those containers are started and identify the pytest fixtures that would own setup and teardown. Done means the required database and integration containers can run through regular tests without Breeze, while Breeze's common dependency container remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, python
Domain
devops, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.