Add tool to re-attempt failed bulk index payloads
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
When crawler fails all retry attempts when bulk indexing a payload into Elasticsearch, it will save the payload as a file to the directory `output/failed_payloads//`. This file's existence is logged to the user but no instructions are provided on what to do with it, and no tools exist currently to easily re-index this data.
### Proposed Solution
Two things should be added to address this
- [ ] The file's payload is already perfect for a bulk index operation. The CLI should have a "re-attempt indexing of failed payloads" command that will attempt to index these files.
- [ ] Documentation should be added explaining what to do with these files.
### Alternatives
Some documentation on how to do this manually might be enough, but it would require users to use Elasticsearch API in the shell which might be a barrier for some.
Contributor guide
Assessment
This issue has not been assessed yet.