electric-sql / electric-sql/electric
Unify protobuf libraries to remove protobuf version pin
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Problem
We pin `protobuf` to `~> 0.13.0` in `packages/sync-service/mix.exs` to avoid a duplicate module conflict at `mix release` time. The conflict is between `Google.Protobuf.*` well-known type modules shipped by two different protobuf libraries in our dependency tree:
- **`protobuf`** (>= 0.14.0) — used by `otel_metric_exporter` for OTLP metric/log encoding. Starting with v0.14.0, it bundles `Google.Protobuf.*` modules (previously in the separate `google_protos` package).
- **`protox`** — used by `pg_query_ex` for Postgres parse tree (de)serialization. Its well-known type source files (`lib/google/protobuf/*.ex`) define `Protox.Google.Protobuf.*` container modules, but the `use Protox.Define` macro inside them generates **bare `Google.Protobuf.*` modules** at compile time (via `DefineMessage.define/2` which does `defmodule unquote(msg_schema.name)` where `name:` is `Google.Protobuf.Any`, not `Protox.Google.Protobuf.Any`).
The conflict only manifests during `mix release` (not `mix compile`), because that's when OTP checks for duplicate modules across applications.
**Neither side's `Google.Protobuf.*` modules are actually used by our code** — `otel_metric_exporter`'s generated `.pb.ex` files are namespaced under `OtelMetricExporter.*`, and `pg_query_ex`'s proto schema has no well-known type imports.
## Approaches explored
### 1. Remove conflicting source files in Dockerfile (quick fix)
Add `RUN rm -rf deps/protox/lib/google` between `mix deps.get` and `mix deps.compile` in the Dockerfile. This prevents protox from compiling the well-known type modules, and Mix auto-generates a clean `.app` file without them.
Verified locally: protox compiles fine (32 files instead of 50) and `mix release` passes.
**Pros:** minimal, no upstream changes needed.
**Cons:** only applies to Docker builds; local `mix release` would still fail without a similar step.
### 2. Fork protox and delete `lib/google/`
Fork `ahamez/protox`, remove the `lib/google/` directory, point `pg_query_ex` at the fork.
**Pros:** clean, works everywhere (not just Docker).
**Cons:** fork maintenance burden.
### 3. Post-compile cleanup in mix.exs
Add a Mix alias that removes `Google.Protobuf.*.beam` files from protox's `ebin/` and patches its `.app` file after compilation.
**Pros:** works for all build contexts.
**Cons:** fragile, modifies compiled artifacts.
### 4. Fix protox upstream
Change protox's well-known type schemas to use `Protox.Google.Protobuf.*` as the module `name:` so `DefineMessage` generates prefixed modules. Requires updating `parse.ex` internals and the code generator's handling of well-known type references. This is a breaking change for protox.
**Pros:** proper fix at the source.
**Cons:** breaking change, significant scope, depends on upstream maintainer.
### 5. Unify on a single protobuf library (preferred direction)
Use either `protobuf` or `protox` in **both** `otel_metric_exporter` and `pg_query_ex`, eliminating the conflict entirely.
- **Option A: Migrate `pg_query_ex` to use `protobuf`** — regenerate the proto files with `protoc-gen-elixir` (from the `protobuf` package) instead of `protox`. `pg_query_ex` is our own package so we control this.
- **Option B: Migrate `otel_metric_exporter` to use `protox`** — regenerate the OTLP proto files with `protox`. Also our own package.
**Pros:** eliminates the root cause, no workarounds needed, one fewer dependency in the tree.
**Cons:** requires regenerating proto code and updating encode/decode call sites.
Contributor guide
Assessment
This issue has not been assessed yet.