Questions about the necessity of creating a dictionary in data_utils.py

Hi, recently I am running DLRM training. I do not understnad the necessity of creating a dictionary in data_utils.py after a modulus operation. This is time-expensive. I know the dictionary is used to collect unique values and then assign correpsonding index to it. But why it is necessary to keep this process here? A simple hashing function, like Sigrid hash, or just modulus should be enough.

Start by reading data_utils.py and trace where the modulus operation and dictionary are used during DLRM training. Determine what behavior the dictionary preserves beyond modulus or hashing, and identify the relevant training path or checks needed to compare alternatives. Done means documenting the necessity or agreeing on a scoped replacement with measurable correctness and performance criteria.

Written by the indexing model from the issue text.