datafusion-contrib / datafusion-contrib/datafusion-distributed
[Abstract protocol] Make tests work with gRPC enabled and disabled
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
This requires depends on the following issue to be shipped first:
Some of the current tests assume that there's a gRPC implementation wired up in the SessionContext, and will attempt to communicate between workers using gRPC, either through in-memory duplex channels or localhost.
This is mostly driven by the test harness shipped in test_utils/. However, once we have a full in-memory implementation of the WorkerChannel, we could be wiring it up to the test utils and run tests with fully in-memory comms.
I think there's still value in being able to run the tests with the gRPC implementation, so the best direction that comes to mind is:
- Scope the gRPC-specific test utils under the
grpc/folder. - In the central
test_utils/go to every place that builds a distributed-capable SessionContext and switch between in-memory or gRPC WorkerChannel implementations based on whether thegrpcflag is enabled or not.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
First review issue #576, then inspect the gRPC-specific helpers under grpc/ and the distributed-capable SessionContext construction in test_utils/. Run the test suite with the grpc flag enabled and disabled to identify assumptions about WorkerChannel implementations. Done means both configurations use the appropriate in-memory or gRPC channel and the tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- distributed-systems, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100