hdfsreader OOM
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
java.lang.OutOfMemoryError: Java heap space
at com.csvreader.CsvReader.updateCurrentValue(Unknown Source) ~[javacsv-2.0.jar:na]
at com.csvreader.CsvReader.checkDataLength(Unknown Source) ~[javacsv-2.0.jar:na]
at com.csvreader.CsvReader.readRecord(Unknown Source) ~[javacsv-2.0.jar:na]
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.splitBufferedReader(UnstructuredStorageReaderUtil.java:72) ~[plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.doReadFromStream(UnstructuredStorageReaderUtil.java:288) ~[plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.readFromStream(UnstructuredStorageReaderUtil.java:214) ~[plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:265) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:57) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Exception in thread "taskGroup-1" com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.OutOfMemoryError: Java heap space
at com.csvreader.CsvReader.updateCurrentValue(Unknown Source)
at com.csvreader.CsvReader.checkDataLength(Unknown Source)
at com.csvreader.CsvReader.readRecord(Unknown Source)
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.splitBufferedReader(UnstructuredStorageReaderUtil.java:72)
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.doReadFromStream(UnstructuredStorageReaderUtil.java:288)
at com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil.readFromStream(UnstructuredStorageReaderUtil.java:214)
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:265)
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:57)
at java.lang.Thread.run(Thread.java:748)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the stack trace locations in UnstructuredStorageReaderUtil.java and HdfsReader.java, then inspect the CsvReader dependency referenced by the failure. The issue does not identify a reproducing input, expected memory behavior, or acceptance criteria, so confirm those with maintainers before proceeding. Done should include a verified resolution of the reported Java heap exhaustion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100