intake / intake/intake-esm

Load only those columns which are defined in the catalog description

Open
#462 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

Here is an idea if the underlying Dataframe is based on a `catalog_file` and in case it is opened and loaded with pandas `read_csv`.

**Is your feature request related to a problem? Please describe.**
The catalog file of a catalog can be very large and contain many columns. It quickly fills user memory when the dataframe is loaded.

**Describe the solution you'd like**
pandas `read_csv` allows to set kwarg `usecols`. One could assemble all columns from the description and set it as values. That reduces the memory requirements.

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or tests. Start by locating the catalog_file loading path and its pandas read_csv call; compare the catalog description's defined columns with the columns requested by the resulting dataframe. Done means read_csv receives the appropriate usecols so unrelated catalog columns are not loaded.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.