h2oai / h2oai/datatable

nrows_max parameter in fread is not handled properly

Open
#1,928 0 comments 0 reactions 0 assignees View on GitHub
bug fread low priority performance
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

`nrows_max` is supposed to limit the number of rows to be read from the file. There are several problems with this parameter:
- When determining chunking parameters, we reduce the estimated number of chunks, to account for the smaller number of rows than the whole file. However, the `compute_chunk_boundaries()` still assigns to the last chunk the whole remainder of the file, making it too big.
- The `read_chunk()` function then proceeds to read the entire chunk, not knowing about the limit on the number of rows, since it doesn't know which row number does it start from. This means the function ends up reading the whole remainder of the file, only to discard the extra rows later.
- If the file is corrupted at a certain line, and the user tries the `max_nrows` parameter in order to read the file up to the point of corruption, the error will still be thrown since the exception is thrown in chunk_reader, before we had the chance to verify the row count, and whether the max_nrows limit was already reached...

It is not entirely clear how to solve this, given that the chunk readers have no way of knowing which row number they begin parsing at... This need some more thought.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.