[FEA] Max Frequency threshold in Categorify

Open
#954 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Assessment

This issue has not been assessed yet.

Description

ops

Is your feature request related to a problem? Please describe.
There seems to be a minimum freq_threshold for Categorify to remove long-tail elements but I feel there are also some use-cases to have a maximum freq_threshold where items/users that appear more than a certain number of times are either meaningless / spam / bots. I understand that it won't have the same kind of memory saving (due to the power-laws in recsys datasets) but it's a pretty common operation in tabular/nlp datasets to ignore extremely frequent elements because they don't carry any information.

Describe the solution you'd like
If Categorify had a max_freq threshold as well, it would feel more intuitive.

Describe alternatives you've considered
Currently, this is doable with a groupBy/valueCounts and a filter operation but it would be good to have support for it out of the box.

Dominant language
Python
Stars
1.2k
Forks
149
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-Merlin/NVTabular

All issues in NVIDIA-Merlin/NVTabular

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.