h2oai / h2oai/datatable

`na.rm` or something similar for aggregations

Open
#3,309 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Not sure if there is an existing ticket, so creating this. Adding a `na.rm`(similar to R) or `skipna` (pandas flavour) parameter in aggregations might be useful

I observed this in functions like `first`:

```py
from datatable import dt, f, by, sort
from numpy import nan

DT = dt.Frame([{'A': 1, 'B': 'a', 'C': 5.0},
{'A': 1, 'B': 'a', 'C': nan},
{'A': 2, 'B': None, 'C': 4.0},
{'A': 2, 'B': 'b', 'C': 4.0},
{'A': 2, 'B': 'b', 'C': nan},
{'A': 3, 'B': 'c', 'C': 9.0},
{'A': 3, 'B': None, 'C': nan},
{'A': 3, 'B': 'd', 'C': nan},
{'A': 3, 'B': None, 'C': 9.0},
{'A': 4, 'B': 'e', 'C': 8.0},
{'A': 4, 'B': 'e', 'C': nan},
{'A': 5, 'B': None, 'C': 2.0},
{'A': 6, 'B': 'h', 'C': nan},
{'A': 6, 'B': 'h', 'C': nan}])

DT
| A B C
| int64 str32 float64
-- + ----- ----- -------
0 | 1 a 5
1 | 1 a NA
2 | 2 NA 4
3 | 2 b 4
4 | 2 b NA
5 | 3 c 9
6 | 3 NA NA
7 | 3 d NA
8 | 3 NA 9
9 | 4 e 8
10 | 4 e NA
11 | 5 NA 2
12 | 6 h NA
13 | 6 h NA
[14 rows x 3 columns]

# returns the exact first value, even if it is a null
DT[:, f.B.first(), 'A']

| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 NA
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]

# a workaround is with sort
# this might be limiting in scenarios where you do not need the sort
# or you have other columns and the sorting affects your output/analysis
DT[:, f.B.first(), 'A', sort(f.B, na_position='last')]
| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 b
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]

# a cleaner, more readable option, and hopefully performant would be sth similar:
DT[:, f.B.first(skipna=True), 'A']

| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 b
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]

Another option might be to change the semantics for first/last to show non-null values if they exist.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.