apache / apache/arrow-java

[FlightRPC][Java] Allow using a per-call Arrow allocator

Open
#329 4 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

We've been running into issues with Flight and gRPC leaking direct memory at scale. One thing we'd like to do is have a (child) allocator per DoGet/DoPut call, so we can more accurately track memory usage. We have a candidate implementation that is rather messy, but can be upstreamed as part of flight-grpc.

This also requires changes to _ensure_ all Arrow resources are cleaned up before we notify gRPC that the call has finished.

**Reporter**: [David Li](https://issues.apache.org/jira/browse/ARROW-9586) / @lidavidm
#### PRs and other links:
- [GitHub Pull Request apache/arrow#8265](https://github.com/apache/arrow/pull/8265)
- [GitHub Pull Request apache/arrow#9417](https://github.com/apache/arrow/pull/9417)

**Note**: *This issue was originally created as [ARROW-9586](https://issues.apache.org/jira/browse/ARROW-9586). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the flight-grpc implementation and the two linked pull requests to understand the candidate approach. Verify that each DoGet/DoPut call can use a child Arrow allocator and that all Arrow resources are cleaned up before gRPC is notified that the call has finished.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.