Optimize lib for large datasets

**Description**
Currently config lacks proper filtering support.

**Proposed solution**
Add configurable filtering with sensible defaults.

**Priority**: Low/Medium

No file, test, or entry point is named. Start by locating the configuration handling and the library path responsible for processing large datasets, then determine how filtering is currently absent. Done means configurable filtering exists with sensible defaults and is covered by relevant tests.

Written by the indexing model from the issue text.