kestra-io / kestra-io/plugin-jdbc

chore: explore Testcontainers migration to reduce heavy Docker containers in CI

Open
#912 0 comments 0 reactions 1 assignee Claimed by @Malaydewangan09 View on GitHub
area/plugin kind/cooldown
Dominant language
Java
Stars
24
Forks
38
Avg merge
2d 3h
Merged PRs (30d)
22

Description

## Summary

The current CI setup (`docker-compose-ci.yml` + `.github/setup-unit.sh`) spins up a large number of heavy Docker containers for unit tests: MySQL, MariaDB, two PostgreSQL instances, ClickHouse, Oracle XE, SQL Server, Trino, Pinot, Dremio, and a full 6-container Apache Druid cluster. This consumes significant GitHub Actions runner disk space, slows down CI setup, and increases flakiness due to container startup timing issues.

This cooldown explores which of those containers can be replaced by [Testcontainers](https://testcontainers.com/) (programmatic, on-demand containers per test class), while being careful **not** to break sanity checks that may legitimately depend on the `docker-compose-ci.yml` services.

## Motivation

- GH Actions runners have limited disk space; running 15+ containers concurrently strains resources.
- The `setup-unit.sh` script has complex wait-for logic with long timeouts (up to 900s for Oracle), making CI slow even when tests pass.
- Testcontainers would start only the containers actually needed by the tests being run, reducing resource usage for partial/targeted test runs.
- Migrating to Testcontainers where possible is already an explicit convention in the project's coding standards.

## Expected Impact

**Current baseline**: CI takes ~21 minutes end-to-end.

The `setup-unit.sh` script runs entirely before the test phase and includes:
- A `docker compose up --wait` for all services (network + image pull + startup time)
- Explicit `wait_for` polls with long timeouts: Oracle up to 900s, each Druid component up to 600s, Pinot up to 420s
- A hard `sleep 20` + Oracle restart after the wait

This upfront setup phase is estimated at **5–8 minutes** of the total 21-minute run on a warm runner (longer on a cold one with image pulls).

With Testcontainers, containers start lazily per test class and the global setup phase disappears entirely for migrated services. Expected outcome:

| Scenario | Estimated CI time | Saving |
|---|---|---|
| Easy wins only (MySQL, MariaDB, Postgres, SQL Server, ClickHouse) | ~16–17 min | ~4–5 min |
| Easy wins + Oracle (if `OracleContainer` works) | ~14–15 min | ~6–7 min |
| Full migration (all services moved or removed) | ~13–14 min | ~7–8 min |

The actual saving will depend on how many services can be migrated without breaking sanity checks (see caveat below).

## Context

Current services declared in `docker-compose-ci.yml` and started by `.github/setup-unit.sh`:

| Service | Port(s) | Notes |
|---|---|---|
| `mysql` | 64790 | Standard MySQL 8 |
| `mariadb` | 64791 | MariaDB 11 with `auth_ed25519` plugin configured post-start |
| `postgres-multi-query` | 56983 | `postgres:latest`, used for multi-query tests |
| `postgres` | 56982 | `postgres:16` with full TLS (custom CA, server cert, client cert, custom `postgresql.conf` + `pg_hba.conf`) |
| `clickhouse` | 28123/29000 | ClickHouse with access management enabled |
| `oracle` | 49161 | Oracle XE 11g — needs post-start DDL + restart to raise `processes`/`sessions` limits |
| `sqlserver` | 41433 | MSSQL Server 2019 |
| `trino` | 48080 | Trino (no extra config) |
| `pinot` | 49000 | Apache Pinot batch quickstart |
| `dremio` | 9047/31010/45678 | Dremio OSS |
| `druid_*` (×6) | various | Full Druid cluster: postgres meta, zookeeper, coordinator, broker, historical, middlemanager, router |

> ⚠️ **Sanity checks caveat**: some of these services may be used by sanity-check flows (`src/test/resources/sanity-checks/`) run via `RunnerTest`. Before migrating any service, verify whether it appears in sanity check tests — those tests cannot rely on Testcontainers and must keep their Docker Compose service or be excluded from the migration scope.

## Suggested Investigation Steps

1. **Audit which services are referenced in sanity checks** — scan `src/test/resources/sanity-checks/` and `*RunnerTest.java` files across all submodules to identify services that must remain in `docker-compose-ci.yml`.
2. **Identify easy wins** — services with a well-supported Testcontainers module and no special post-start setup are the best candidates:
- `mysql` → [`MySQLContainer`](https://java.testcontainers.org/modules/databases/mysql/)
- `mariadb` → [`MariaDBContainer`](https://java.testcontainers.org/modules/databases/mariadb/) (investigate whether `auth_ed25519` can be configured via init script)
- `postgres-multi-query` → [`PostgreSQLContainer`](https://java.testcontainers.org/modules/databases/postgres/)
- `sqlserver` → [`MSSQLServerContainer`](https://java.testcontainers.org/modules/databases/mssql/)
- `clickhouse` → [`ClickHouseContainer`](https://java.testcontainers.org/modules/clickhouse/)
3. **Investigate harder cases** — services with complex startup or no official Testcontainers module:
- `postgres` (TLS) — evaluate whether `PostgreSQLContainer` can be configured with custom SSL certs via init scripts or container customization.
- `oracle` — evaluate `OracleContainer` (requires accepting the Oracle license); assess whether the `processes`/`sessions` DDL workaround can be embedded.
- `druid` (6 containers) — no official Testcontainers module; likely keep in Docker Compose or skip.
- `pinot`, `dremio`, `trino` — check for community Testcontainers modules.
4. **Migrate per submodule** — for each service that can be migrated, update the test class(es) in the corresponding submodule, remove the service from `docker-compose-ci.yml`, and simplify the `setup-unit.sh` wait-for logic accordingly.
5. **Verify CI** — ensure no sanity checks are broken and that the runner disk usage improves.

## Acceptance Criteria

- [ ] Audit of sanity-check vs unit-test service usage is documented (can be in a PR description or as inline comments in `docker-compose-ci.yml`)
- [ ] At least the services with official Testcontainers modules and no sanity-check dependency are migrated
- [ ] Migrated submodules no longer require their corresponding Docker Compose service to be running for `./gradlew test`
- [ ] `docker-compose-ci.yml` and `setup-unit.sh` are updated to remove migrated services
- [ ] CI passes with the reduced container set
- [ ] Sanity-check tests are unaffected

## References

- [Testcontainers Java — Database modules](https://java.testcontainers.org/modules/databases/)
- Current `docker-compose-ci.yml` and `.github/setup-unit.sh` in this repository

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.