apache / apache/logging-flume

flume-hdfs-sink Transmission speed

Open
#424 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.6k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
7

Description

For each event, a task will be submitted to the callTimeoutPool thread pool during append, and then the future.get method will be used to obtain the result. This may cause a large load on the thread pool, which cannot be processed at all. Through flame graph analysis, it can also be seen that retrieving tasks from the thread pool takes a lot of time. Can we change it to synchronous

Image

Contributor guide

Open the contributing guide

Research direction

Start in the flume-hdfs-sink append path, focusing on callTimeoutPool task submission and the subsequent future.get call. Review the flame graph and current behavior, then determine whether synchronous handling is appropriate; done should address the reported thread-pool load and transmission-speed concern, with validation based on the observed performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.