intake / intake/intake-esm

Search esm-datastores using polars

Open
#711 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

Following loading a datastore using polars, we can run queries against it using polars, which should be significantly faster than the current pandas based approaches.

**Describe the solution you'd like**

Change the search functionality in `_search.py` to use polars.

**Additional context**

Sub issue of #705 . Requires #709 .

Contributor guide

Open the contributing guide

Research direction

Start in _search.py and review the current pandas-based search functionality, along with the context from prerequisite issue #709 and parent issue #705. Update the search path to use polars while preserving the existing query behavior, then verify that datastore searches work through the polars-based path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.