dssg / dssg/triage

Helpful ideas for feature validation

Open
#514 0 comments 0 reactions 2 assignees Claimed by @thcrock View on GitHub
Dominant language
Jupyter Notebook
Stars
201
Forks
62
PR merge metrics
No merged PRs in 30d

Description

I recently stumble several times with this error in triage:

```
ValueError: Missing columns in get_impute_create: {'demographics_entity_id_all_primarypayercode_country_indigent_max,
}
```

And I noticed that those names have longer lengths that the ones admitted for column names in PostgreSQL. Since `collate` is looking up the column names in the `features_aggregation` table by creating a `{feature_group}_{group}_{column_name}_{aggregation_fx}` naming, it's not able to grab the truncated names that PostgreSQL is using. In my case: `demographics_entity_id_all_primarypayercode_countr`.

I would add a feature into the validator to make a column name length calculation using the elements from the `feature_aggregation` elements in the triage's configuration file. We can use the `choices` or the `choice_query` to grab these errors before running triage. Is this a good idea? I'm the only one with long name problems?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.