bcgov / bcgov/diputils

2. basic ops

Open
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Basic operations to support minimal data quality assessment, make life more live-able, and increase the ease and effectiveness for data-science swat-team deployments, all in the large-tabular-data context

* path normalization for interop between environments (classify path format by OS and translate to native format)
* data type detect: nominal, numeric, date, geo
* date detect and format validation
* data dictionary vs file matching
* data dict normalization plus recovery from multiline cells
* metadata: fields search, description search, w support for fuzzy matching
* semantic matching
* autodetect and application of human-readable lookups present in other tables
* flatfile parsing -- all sets
* dataset identification and integration
* redundant records detection -- large data
* lossless data compression
* windowing for multitemporal analysis
* low memory (large data) sorting, incl. but not limited to: by date!
* not require specific install location
* allow people to select versions for data
* parse and filter largest files bypassing RAM memory limitation restrictions

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.