CI: ship connector plugins in the `apache/iggy-connect` docker image
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
the `apache/iggy-connect` image ships only the `iggy-connectors` runtime binary. no plugin `.so` files inside. so anyone who wants to actually try a connector has to get the plugins from somewhere else first: 9 of our 15 plugins are prebuilt in the edge github pre-release tarballs (download, unpack, mount into the container), and the remaining 6 (delta, doris, http, influxdb sink + source, mongodb) are not prebuilt anywhere - those require cloning the repo and a full rust toolchain. either way it is a lot of friction for what should be a five minute "docker run + config file" experience.
## proposal
two flavors of the image, distinguished by tag suffix (like `python:3.12-slim`):
| tag | contents |
| --- | --- |
| `edge`, `x.y.z`, `latest` | runtime + all connector plugins baked in (new default) |
| `edge-slim`, `x.y.z-slim`, `latest-slim` | runtime binary only (what the image is today) |
the fat image becomes the default because that is what a new user expects: pull, point a config at a connector, go. size-sensitive deployments that bring their own plugins switch to `-slim`.
plugins land in `/usr/local/lib`, which is already on the runtime's plugin search path - so a connector config can reference just `libiggy_connector_postgres_sink` with no absolute path. zero-config out of the box.
## what gets bundled
all 15 cdylib plugins currently in the workspace:
- sinks: delta, doris, elasticsearch, http, iceberg, influxdb, mongodb, postgres, quickwit, s3, stdout
- sources: elasticsearch, influxdb, postgres, random
side note: the edge tarball plugin list in `_build_rust_artifacts.yml` is a hardcoded default with only 9 of these, and the packaging step silently skips a missing `.so` instead of failing. the plugin list should come from one place so new connectors stop missing release artifacts - pre-merge CI already derives bin + cdylib packages from `cargo metadata`, and the cdylib filter alone is exactly this set, so there is prior art to reuse.
## how (roughly)
- `core/connectors/runtime/Dockerfile`: the cargo-chef cook and build steps currently target `-p iggy-connectors` only, so the heavy plugin dependencies (deltalake, arrow, aws sdk, mongodb driver...) are not in the cached recipe today. both steps grow the plugin packages, plus a second final stage that COPYs the `.so` files into `/usr/local/lib`. slim stage stays exactly what it is today.
- `.github/actions/utils/docker-buildx` action: needs to learn to build a specific dockerfile stage (`target:` is not plumbed through today), or `.github/config/publish.yml` grows a second component entry per flavor (which also means touching the hardcoded docker component allowlist in the publish plan job).
- `docker-manifests` job: pushes only `edge` / `x.y.z` / `latest` today, so the `-slim` variants are new. the real work is the digest plumbing: per-arch digests travel as `docker-digest--` artifacts and the manifest is created by globbing every digest in one directory, so the two flavors must keep separate digest sets or they merge into one manifest.
- edge gate: `rust-connectors` currently gates on `crates: [iggy-connectors]` only (plugins are deliberately excluded since the image does not ship them - the config comment says so). with plugins bundled, the gate expands to the full connector crate set, and that comment gets updated.
- licensing: the image generates `LICENSE-binary` from `core/connectors/runtime/Cargo.toml` only. `third-party-licenses.sh` already accepts multiple `--manifest` flags and merges rust closures, so the generate step in the dockerfile and the "validate iggy-connect bundle" step in the validate action both grow the 15 plugin manifests. heads up: the plugin dependency closures have never been license-validated anywhere, so the first run may surface ASF category-x findings that block bundling a plugin until resolved.
## tradeoffs
- image size grows by 15 statically-linked `.so` files (arrow, deltalake, aws sdk closures - and the release profile does not strip symbols today, only default debuginfo stripping). needs an actual measurement before merging, plus a strip step if the numbers are silly. `-slim` exists for anyone who cares.
- docker build time goes up materially: the plugin deps compile cold until the chef recipe includes them, CI caching is layer-level (any workspace dependency change re-cooks everything, per arch, with lto + codegen-units=1), and the publish job has a 60 minute timeout. worth watching on the first runs.
- tag semantics change: `:edge` starts pulling fat on the next master push after merge, `:latest` and versioned tags only from the next release onward. existing users get a bigger image with identical behavior (plugins are inert files until a config loads them), but release notes and the docker hub description must call it out.
- bigger dependency surface on the default image means CVE scanners will flag plugin closures for every user, including ones who never load a plugin. `-slim` is the answer, but expect the reports.
## Tasks
- [ ] dockerfile: extend chef cook + build to plugin crates, fat final stage with `.so` in `/usr/local/lib`
- [ ] publish pipeline: `-slim` flavor end to end - buildx action (stage target or second component entry + plan-job allowlist), per-flavor digest artifacts, `-slim` manifests
- [ ] expand edge gate to plugin crates, update the stale config comment
- [ ] unify the plugin list (docker image, edge tarballs, release notes) via the existing `cargo metadata` cdylib derivation; make missing `.so` a hard failure in tarball packaging
- [ ] extend `LICENSE-binary` generation and the license validation gate to plugin manifests; resolve whatever category-x findings show up
- [ ] measure fat image size, add strip if needed; verify the heavy plugin closures cross-compile under the alpine + zig builder
- [ ] docs: connectors README, docker hub description, quickstart example using the bundled plugins
Contributor guide
Assessment
This issue has not been assessed yet.