Make purge crawl pagination lazy
Open
complexity:medium
enhancement
priority:low
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
Pagination for the purge crawl feature isn't paginated, so all IDs are stored in memory during the sync. This could cause problems for users that need to purge very large amounts of docs.
### Proposed Solution
Make the pagination lazy so we can avoid this.
### Additional Context
Came up in this PR: https://github.com/elastic/crawler/pull/65#discussion_r1711771484
Contributor guide
Assessment
This issue has not been assessed yet.