Optimize lib for large datasets

**Description**
Currently core lacks proper compression support.

**Proposed solution**
Add configurable compression with sensible defaults.

**Priority**: Low/Medium

No file, test, or entry point is named. Start by locating the core data path and checking how large datasets are currently handled, then identify the supported compression options and configuration surface. Done means configurable compression with sensible defaults and verified behavior for large datasets.

Written by the indexing model from the issue text.