能否实现单纯的文件传输插件,例如ftp - hdfs
Open
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
1.因为有这样的需求。尝试自定义插件,使用ByteColumn去传递数据。但是这种情况很容易内存溢出,而且还发现了部分传输之后文件失败的情况,是否应该split针对大文件切分?
2.看到是有streamreader和streamwirter但是没有对应md了 这两种插件是否可以使用流来传输?
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin with the existing streamreader and streamwirter plugins and inspect how ByteColumn currently carries file data. Clarify whether an FTP-to-HDFS transfer should stream, how large files should be split, and how partial-transfer failures are handled; done means an agreed plugin scope and documented usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100