h2oai / h2oai/datatable

xread: an "extract+read"

Open
#1,950 1 comment 0 reactions 0 assignees View on GitHub
cust-goldmansachs design-doc fread
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

This is a proposal for implementing a new function `xread()`, which would be conceptually similar to `fread()`, but much lazier. In particular, `xread()` would parse only the first `n_sample_lines=100` lines of the file, detecting the general information such as parse settings, the number columns, their names and types. After that, `xread()` returns a "lazy frame" object, which can be used with the standard `[i,j]` notation:

- `[:n, :]` returns just the first `n` rows of the dataset (equivalent to `max_nrows` parameter);
- `[1000:2000, :]` returns rows from 1000 to 2000. Generally, we should allow the user to request consecutive ranges on the same lazy frame. This will be equivalent to "chunked reading", which is a popular request;
- `[-100:, :]` returns the last 100 rows. For a "file object" sources, this would require that the file is read in its entirety. For files on disk (or in memory), we could try to parse from the end of the file.
- `[:, :5]` return only the first 5 columns.
- `[:, ["A", "B", "C"]]` return columns named `A`, `B` and `C`.
- `[:, f + {"wage_per_hour": f.salary/f.hours}]` return all columns + an additional column containing the `salary` divided by `hours`.
- `[f.status != '', :]` return only the rows where field `status` is not empty.
- `[dt.random.randu() < 0.2, :]` randomly sample 20% of the rows.
- ...

These are just some of the examples of what could be possible. Obviously, the `i` and `j` selectors can be combined into a single `[i, j]` selector too. It should be even possible to add join operations and groupbys into the mix (provided that we use single-pass hash-based grouping).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.