[QST] eager statistics execution and lazy transformation execution

Start with Workflow.add_cat_preprocess, LambdaOp, Categorify, and the _top_level_groupby traceback. Reproduce the two crossed columns using the provided workflow and inspect when LambdaOp output is made available to Categorify statistics. Done means the reported KeyError is resolved or the eager-statistics behavior and required workflow ordering are documented.

What is your question?

Hi, the NVTabular 0.2 said that getting statistics is a eager execution.

For a pure Workflow, this means the data will only be read into memory when statistics are needed to complete the construction of the full task graph, or when a partition needs to be persisted to disk

I am a bit confused, since sometimes statistics need the completion of previous transformation. As a result, it should the case that: when statistics are needed, the previous computation will first be completed ? am I right?

Another thing is that is add_cont_preprocess/feature, and add_cat_preprocess/feature is independent of each other ?

I got errors when I want to add two new crossed cols ("C1_C2, C3_C4"). The errors show that the LambdaOp was not executed when it wants to get some statistics of the crossed cols. It really confused me.

my code

proc = nvt.Workflow(cat_names=CATEGORICAL_COLUMNS,cont_names=CONTINUOUS_COLUMNS,label_name=LABEL_COLUMNS,client=client)

proc.add_feature(ops.FillMissing())

CROSS_COLUMNS = []
feature_cross_list  = "C1_C2,C3_C4"
CROSS_COLUMNS = []
## Feature Crossing of C1 and C2, C3 and C4
if feature_cross_list:
    feature_pairs = [pair.split("_") for pair in feature_cross_list.split(",")]
    for pair in feature_pairs:
        col0 = pair[0]
        col1 = pair[1]
        col0_1 = col0+'_'+col1
        CROSS_COLUMNS.append(col0+'_'+col1)
        ## LambdaOp will automatically add new column with the name of col_name + "_" + op_name for differentiation
        proc.add_cat_preprocess(ops.LambdaOp(op_name=col1,f=lambda col, gdf: col + gdf[col1], columns=[col0], replace=False))

freq_limit = 6
proc.add_cat_preprocess(ops.Categorify(freq_threshold=freq_limit, columns = CATEGORICAL_COLUMNS + CROSS_COLUMNS, out_path=stats_path))

error messages

Function:  _top_level_groupby
args:      (             I1      I2    I3     I4     I5   I6   I7  ...          C21         C22          C23         C24         C25         C26 label 
0           3.0   112.0   2.0   30.0    6.0  0.0  0.0  ...   1190262723  1528360435   -573283542   265389908 -1726799382  -317696227   0.0
1           6.0   292.0  11.0  156.0    5.0  0.0  0.0  ...  -1475649974  -232742063   1228114737  -836282684   809724924 -1604235575   0.0
2           4.0    <NA>   3.0    3.0   <NA>  7.0  1.0  ...  -1761877609   357969245   -740331133   638354314   809724924  -507617550   0.0
3          35.0    30.0   5.0   36.0   19.0  0.0  0.0  ...  -1761877609   357969245   -740331133  1646760685   809724924  -507617550   0.0
4           3.0   306.0  10.0   69.0   12.0  0.0  0.0  ...   1546299542  1542038664  -1378100827   363670520   -10139646 -1218975401   0.0
...         ...     ...   ...    ...    ...  ...  ...  ...          ...         ...          ...         ...         ...         ...   ... 
15266211    8.0     6.0
kwargs:    {}
Exception: KeyError('C1_C2')

distributed.worker - WARNING -  Compute Failed 
Function:  _top_level_groupby
args:      (             I1     I2    I3     I4    I5    I6   I7  ...         C21         C22          C23         C24         C25         C26  label  
0          44.0  698.0   9.0  148.0   3.0   0.0  0.0  ...  -767254031  -126473935   -267996249  1394772638  2020056016   751740247    0.0
1           8.0  145.0  26.0   <NA>  <NA>   0.0  0.0  ...   109296751   369154330         <NA>  -905710383   809724924   751740247    0.0
2           6.0  144.0   1.0   <NA>  <NA>   0.0  0.0  ...  -754672939 -1728259433  -1081340515  -653876797   809724924 -1218975401    0.0
3         101.0  219.0  10.0  213.0  26.0   0.0  1.0  ...   618888848  1611105791    705649113  -788690383  -842849922  -317696227    0.0
4           3.0  767.0   7.0   31.0  99.0   0.0  0.0  ... -1762155011 -2127448782         <NA>  1969938350  1474520440 -1218975401    1.0
...         ...    ...   ...    ...   ...   ...  ...  ...         ...         ...          ...         ...         ...         ...    ...
15266211    1.0   70.0  <NA>  109
kwargs:    {}
Exception: KeyError('C1_C2')