Remove Protobuf runtime dependency from OTLP exporter

Is your feature request related to a problem?

Protobuf is a heavy package (~2MB) and the strict version constraint can be burdensome to manage when adding OTLP exporter to applications that already depend on Protobuf.

Describe the solution you'd like

I came across this post from opentelemetry-java Sharing proto related techniques from opentelemetry-java #1996 describing the success the team had from removing the dependency regarding package sizes and serialization performance. Also support for Protobuf 3 and 4 will be dropped in the next release of opentlemetry-python to support Protobuf 5 Add support for Protobuf 5 #3958. Going forward Protobuf will having a rolling compatibility window of 2 versions. However, opentelemetry-python could still benefit from removing Protobuf as a runtime dependency entirely in favor of custom serialization logic into Proto wire format like opentelemetry-java has done:

• Faster package loading (10ms faster if wheels available for google.protobuf, up to 100ms faster if not)
• Reduced package sizes (~2MB)
• Better serialization performance (Slightly faster, significantly less memory. exact numbers TBD)
• Eliminate risk of conflicts in user applications that depend on Protobuf

The immediate downside is that it will add some extra code for the serialization logic that needs to be maintained in place of Protobuf. I would argue that the maintenance burden is small, as the Proto wire format is guaranteed to always be backwards compatible, regardless of if the google.protobuf package itself breaks compatibility with prior versions.

Describe alternatives you've considered

• Internalizing Protobuf library to avoid the external runtime dependency
• Pregenerating pb2 files for all major Protobuf versions and dynamically loading based on runtime version like wandb

Both these solutions do not solve the issue of loading times or performance. The first solution would require a custom approach since Python does not support shading dependencies. The second solution would also require some additional work and testing for every new major Protobuf release and could be difficult to manage backwards compatibility.

Additional Context

I am working on a POC to remove Protobuf dependency from the OTLP exporter for an internal fork/wrapper that will allow us to enable OTLP exporting of telemetry data in Python runtimes without having to worry about Protobuf version conflicts at present or in the future. We are implementing custom serialization logic similar to the opentelemetry-java, and are expecting to see the aforementioned performance benefits. I wanted to share this with the community if you think upstreaming this work would be beneficial. If so, I would be happy to share more information on my design and receive some feedback.

Would you like to implement a fix?

Yes

Start with the OTLP exporter and review the linked opentelemetry-java discussion and Protobuf 5 issue for compatibility context. Compare the proposed custom serialization approach with the existing internal POC, then define the required wire-format coverage and performance checks before implementation. Done means the exporter no longer requires Protobuf at runtime without breaking supported telemetry serialization.