elastic / elastic/crawler

Make purge crawl pagination lazy

Open
#102 0 comments 0 reactions 0 assignees View on GitHub
complexity:medium enhancement priority:low
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

### Problem Description

Pagination for the purge crawl feature isn't paginated, so all IDs are stored in memory during the sync. This could cause problems for users that need to purge very large amounts of docs.

### Proposed Solution

Make the pagination lazy so we can avoid this.

### Additional Context

Came up in this PR: https://github.com/elastic/crawler/pull/65#discussion_r1711771484

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.