alibaba / alibaba/DataX

能否实现单纯的文件传输插件,例如ftp - hdfs

Open
#632 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

1.因为有这样的需求。尝试自定义插件,使用ByteColumn去传递数据。但是这种情况很容易内存溢出,而且还发现了部分传输之后文件失败的情况,是否应该split针对大文件切分?

2.看到是有streamreader和streamwirter但是没有对应md了 这两种插件是否可以使用流来传输?

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with the existing streamreader and streamwirter plugins and inspect how ByteColumn currently carries file data. Clarify whether an FTP-to-HDFS transfer should stream, how large files should be split, and how partial-transfer failures are handled; done means an agreed plugin scope and documented usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.