datafusion-contrib / datafusion-contrib/datafusion-distributed

[Abstract protocol] Make tests work with gRPC enabled and disabled

Open
#579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

This requires depends on the following issue to be shipped first:

Some of the current tests assume that there's a gRPC implementation wired up in the SessionContext, and will attempt to communicate between workers using gRPC, either through in-memory duplex channels or localhost.

This is mostly driven by the test harness shipped in test_utils/. However, once we have a full in-memory implementation of the WorkerChannel, we could be wiring it up to the test utils and run tests with fully in-memory comms.

I think there's still value in being able to run the tests with the gRPC implementation, so the best direction that comes to mind is:

  1. Scope the gRPC-specific test utils under the grpc/ folder.
  2. In the central test_utils/ go to every place that builds a distributed-capable SessionContext and switch between in-memory or gRPC WorkerChannel implementations based on whether the grpc flag is enabled or not.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

First review issue #576, then inspect the gRPC-specific helpers under grpc/ and the distributed-capable SessionContext construction in test_utils/. Run the test suite with the grpc flag enabled and disabled to identify assumptions about WorkerChannel implementations. Done means both configurations use the appropriate in-memory or gRPC channel and the tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
distributed-systems, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.