`na.rm` or something similar for aggregations
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Not sure if there is an existing ticket, so creating this. Adding a `na.rm`(similar to R) or `skipna` (pandas flavour) parameter in aggregations might be useful
I observed this in functions like `first`:
```py
from datatable import dt, f, by, sort
from numpy import nan
DT = dt.Frame([{'A': 1, 'B': 'a', 'C': 5.0},
{'A': 1, 'B': 'a', 'C': nan},
{'A': 2, 'B': None, 'C': 4.0},
{'A': 2, 'B': 'b', 'C': 4.0},
{'A': 2, 'B': 'b', 'C': nan},
{'A': 3, 'B': 'c', 'C': 9.0},
{'A': 3, 'B': None, 'C': nan},
{'A': 3, 'B': 'd', 'C': nan},
{'A': 3, 'B': None, 'C': 9.0},
{'A': 4, 'B': 'e', 'C': 8.0},
{'A': 4, 'B': 'e', 'C': nan},
{'A': 5, 'B': None, 'C': 2.0},
{'A': 6, 'B': 'h', 'C': nan},
{'A': 6, 'B': 'h', 'C': nan}])
DT
| A B C
| int64 str32 float64
-- + ----- ----- -------
0 | 1 a 5
1 | 1 a NA
2 | 2 NA 4
3 | 2 b 4
4 | 2 b NA
5 | 3 c 9
6 | 3 NA NA
7 | 3 d NA
8 | 3 NA 9
9 | 4 e 8
10 | 4 e NA
11 | 5 NA 2
12 | 6 h NA
13 | 6 h NA
[14 rows x 3 columns]
# returns the exact first value, even if it is a null
DT[:, f.B.first(), 'A']
| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 NA
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]
# a workaround is with sort
# this might be limiting in scenarios where you do not need the sort
# or you have other columns and the sorting affects your output/analysis
DT[:, f.B.first(), 'A', sort(f.B, na_position='last')]
| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 b
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]
# a cleaner, more readable option, and hopefully performant would be sth similar:
DT[:, f.B.first(skipna=True), 'A']
| A B
| int64 str32
-- + ----- -----
0 | 1 a
1 | 2 b
2 | 3 c
3 | 4 e
4 | 5 NA
5 | 6 h
[6 rows x 2 columns]
Another option might be to change the semantics for first/last to show non-null values if they exist.
```
Contributor guide
Assessment
This issue has not been assessed yet.