alibaba / alibaba/DataX

HdfsReader的线程数量,通过通道数指定无效,是根据文件数量决定的。

Open
#901 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

文档中写到"HdfsReader支持使用多线程进行数据抽取。线程并发数通过通道数指定。"

![image](https://user-images.githubusercontent.com/16739927/100991610-28bc7680-358e-11eb-83eb-021670c6bd72.png)

但是看代码,是根据文件数量决定的。(后续引起了TxtFileWriter生成了多个文件)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the HdfsReader implementation and compare its thread-count logic with the documentation statement quoted in the issue. Reproduce the behavior with different file and channel counts, then make the channel-count contract consistent and verify that TxtFileWriter no longer receives unintended parallel output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.