Load only those columns which are defined in the catalog description
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
Here is an idea if the underlying Dataframe is based on a `catalog_file` and in case it is opened and loaded with pandas `read_csv`.
**Is your feature request related to a problem? Please describe.**
The catalog file of a catalog can be very large and contain many columns. It quickly fills user memory when the dataframe is loaded.
**Describe the solution you'd like**
pandas `read_csv` allows to set kwarg `usecols`. One could assemble all columns from the description and set it as values. That reduces the memory requirements.
Contributor guide
Research direction
The issue names no repository files or tests. Start by locating the catalog_file loading path and its pandas read_csv call; compare the catalog description's defined columns with the columns requested by the resulting dataframe. Done means read_csv receives the appropriate usecols so unrelated catalog columns are not loaded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100