[Task] Using DataFrame level sum, count, min, max, std, var functions in Ops
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start by locating the Ops that iterate over dataframe columns for sum, count, min, max, std, var, or mean, then read the corresponding cuDF DataFrame aggregation documentation linked in the issue. Done means the relevant Ops use dataframe-level functions rather than per-column calls, with existing behavior preserved and any available tests or benchmarks checked.
Written by the indexing model from the issue text.
Description
Monday Jun 01, 2020 at 15:07 GMT
Originally opened as https://github.com/rapidsai/recsys/issues/188
Right now in nvtabular, we go over all the columns of a dataframe one by one to run the functions like sum, count, min, max, etc.
We made feature requests for these functions to be available in dataframe level and cudf team let me know that they are available in branch-0.14.
Here are the doc links for these functions;
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.sum
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.count
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.max
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.min
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.var
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.std
https://docs.rapids.ai/api/cudf/nightly/api.html#cudf.core.dataframe.DataFrame.mean
The Ops that are using these functions should be updated to call these functions in the dataframe level. Hoping that these dataframe level functions are implemented in an optimal way so that we can see some speedup in our ops.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-Merlin/NVTabular
-
[DOC] Open
Difficulty 4/5 3-5 days Newbie friendliness 20/100
NVIDIA-Merlin/NVTabular#1898 · 1 comment ·
-
question
Difficulty 5/5 Over a week Newbie friendliness 25/100
NVIDIA-Merlin/NVTabular#1895 ·
-
question
Difficulty 4/5 3-5 days Newbie friendliness 25/100
NVIDIA-Merlin/NVTabular#1893 ·
-
question
Difficulty 5/5 Over a week Newbie friendliness 20/100
NVIDIA-Merlin/NVTabular#1892 · 1 reaction ·
-
question
Difficulty 4/5 3-5 days Newbie friendliness 35/100
NVIDIA-Merlin/NVTabular#1891 ·
All issues in NVIDIA-Merlin/NVTabular
Similar issues
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100