[Task]: Beam tft extra depends on old version of tensorflow-transform that requires protobuf v4.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What needs to happen?
We recently started to allow newer versions of GRPC. https://github.com/apache/beam/pull/37817.
I wanted to regenerate containers to start using newer versions of GRPC and clean up some tech debt we had in our code because grpc version was restricted, notably envoy-data-plane was set to use some really old version.
I ran into dependency hell in some of our test suites. The crux of it seems to be:
- our ml_test dependency requires tensorflow-transform
- tensorflow-transform doesn't support protobuf 6
- beam tft extra requires an even older tensorflow-transform that doesn't support protobuf 5. Our ml_tests suites (e.g. precommit XLang YAML) end up using protobuf 3 due to various dependency constraints.
- grpcio-status==1.78.0 requires protobuf 6: protobuf 5 will reach EOL at the end of March'26 .
- If we build Beam GRPC stubs with protobuf 5+, these stubs no longer work with protobuf 3: https://github.com/apache/beam/pull/37822/changes/531176cbbfe680e688567949f34ebc320a17074e#r2932955415, in other words install pip install apache-beam[tft] will not work.
We need to upgrade to newer versions of TFT, which is currently blocked by https://github.com/tensorflow/transform/issues/347 .
### Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
No specific files are named; start by locating the Beam tft extra and the ml_test dependency definitions, then reproduce the conflict with `pip install apache-beam[tft]` and the precommit XLang YAML suites. Done means newer TensorFlow Transform dependencies work with the required protobuf and gRPC versions without breaking those tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- build-system, machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100