electric-sql / electric-sql/electric

Unify protobuf libraries to remove protobuf version pin

Open
#4,018 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Problem

We pin `protobuf` to `~> 0.13.0` in `packages/sync-service/mix.exs` to avoid a duplicate module conflict at `mix release` time. The conflict is between `Google.Protobuf.*` well-known type modules shipped by two different protobuf libraries in our dependency tree:

- **`protobuf`** (>= 0.14.0) — used by `otel_metric_exporter` for OTLP metric/log encoding. Starting with v0.14.0, it bundles `Google.Protobuf.*` modules (previously in the separate `google_protos` package).
- **`protox`** — used by `pg_query_ex` for Postgres parse tree (de)serialization. Its well-known type source files (`lib/google/protobuf/*.ex`) define `Protox.Google.Protobuf.*` container modules, but the `use Protox.Define` macro inside them generates **bare `Google.Protobuf.*` modules** at compile time (via `DefineMessage.define/2` which does `defmodule unquote(msg_schema.name)` where `name:` is `Google.Protobuf.Any`, not `Protox.Google.Protobuf.Any`).

The conflict only manifests during `mix release` (not `mix compile`), because that's when OTP checks for duplicate modules across applications.

**Neither side's `Google.Protobuf.*` modules are actually used by our code** — `otel_metric_exporter`'s generated `.pb.ex` files are namespaced under `OtelMetricExporter.*`, and `pg_query_ex`'s proto schema has no well-known type imports.

## Approaches explored

### 1. Remove conflicting source files in Dockerfile (quick fix)

Add `RUN rm -rf deps/protox/lib/google` between `mix deps.get` and `mix deps.compile` in the Dockerfile. This prevents protox from compiling the well-known type modules, and Mix auto-generates a clean `.app` file without them.

Verified locally: protox compiles fine (32 files instead of 50) and `mix release` passes.

**Pros:** minimal, no upstream changes needed.
**Cons:** only applies to Docker builds; local `mix release` would still fail without a similar step.

### 2. Fork protox and delete `lib/google/`

Fork `ahamez/protox`, remove the `lib/google/` directory, point `pg_query_ex` at the fork.

**Pros:** clean, works everywhere (not just Docker).
**Cons:** fork maintenance burden.

### 3. Post-compile cleanup in mix.exs

Add a Mix alias that removes `Google.Protobuf.*.beam` files from protox's `ebin/` and patches its `.app` file after compilation.

**Pros:** works for all build contexts.
**Cons:** fragile, modifies compiled artifacts.

### 4. Fix protox upstream

Change protox's well-known type schemas to use `Protox.Google.Protobuf.*` as the module `name:` so `DefineMessage` generates prefixed modules. Requires updating `parse.ex` internals and the code generator's handling of well-known type references. This is a breaking change for protox.

**Pros:** proper fix at the source.
**Cons:** breaking change, significant scope, depends on upstream maintainer.

### 5. Unify on a single protobuf library (preferred direction)

Use either `protobuf` or `protox` in **both** `otel_metric_exporter` and `pg_query_ex`, eliminating the conflict entirely.

- **Option A: Migrate `pg_query_ex` to use `protobuf`** — regenerate the proto files with `protoc-gen-elixir` (from the `protobuf` package) instead of `protox`. `pg_query_ex` is our own package so we control this.
- **Option B: Migrate `otel_metric_exporter` to use `protox`** — regenerate the OTLP proto files with `protox`. Also our own package.

**Pros:** eliminates the root cause, no workarounds needed, one fewer dependency in the tree.
**Cons:** requires regenerating proto code and updating encode/decode call sites.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.