Optimize api for large datasets

**Description**
Currently core lacks proper pagination support.

**Proposed solution**
Add configurable pagination with sensible defaults.

**Priority**: Low/Medium

Start by locating the core API entry point and tracing how large datasets are currently returned. Determine where configurable pagination and its defaults belong, then verify that API responses are paginated consistently and that the configuration works for large datasets.

Written by the indexing model from the issue text.