carbonReader read incomplete data
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 694
- PR merge metrics
- No merged PRs in 30d
Description
Look at this line of code. `boolean hasNext = currentReader.nextKeyValue();`
If hasNext returns false and currentReader is not the last one, it indicates that the iterator exits and subsequent data is not parsed. How to solve this problem?
```
/**
* Return true if has next row
*/
public boolean hasNext() throws IOException, InterruptedException {
if (0 == readers.size() || currentReader == null) {
return false;
}
validateReader();
if (currentReader.nextKeyValue()) {
return true;
} else {
if (index == readers.size() - 1) {
// no more readers
return false;
} else {
// current reader is closed
currentReader.close();
// no need to keep a reference to CarbonVectorizedRecordReader,
// until all the readers are processed.
// If readers count is very high,
// we get OOM as GC not happened for any of the content in CarbonVectorizedRecordReader
readers.set(index, null);
index++;
currentReader = readers.get(index);
boolean hasNext = currentReader.nextKeyValue();
if (hasNext) {
return true;
}
}
}
return false;
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the hasNext method shown in the issue and the CarbonVectorizedRecordReader instances it advances through. Reproduce a case where an intermediate reader returns false, then verify that iteration continues and all subsequent data is parsed before treating the issue as done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100