apache / apache/carbondata

carbonReader read incomplete data

Open
#4,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
694
PR merge metrics
No merged PRs in 30d

Description

Look at this line of code. `boolean hasNext = currentReader.nextKeyValue();`
If hasNext returns false and currentReader is not the last one, it indicates that the iterator exits and subsequent data is not parsed. How to solve this problem?
```
/**

* Return true if has next row
*/
public boolean hasNext() throws IOException, InterruptedException {
if (0 == readers.size() || currentReader == null) {
return false;
}
validateReader();
if (currentReader.nextKeyValue()) {
return true;
} else {
if (index == readers.size() - 1) {
// no more readers
return false;
} else {
// current reader is closed
currentReader.close();
// no need to keep a reference to CarbonVectorizedRecordReader,
// until all the readers are processed.
// If readers count is very high,
// we get OOM as GC not happened for any of the content in CarbonVectorizedRecordReader
readers.set(index, null);
index++;
currentReader = readers.get(index);
boolean hasNext = currentReader.nextKeyValue();
if (hasNext) {
return true;
}
}
}
return false;
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the hasNext method shown in the issue and the CarbonVectorizedRecordReader instances it advances through. Reproduce a case where an intermediate reader returns false, then verify that iteration continues and all subsequent data is parsed before treating the issue as done.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.