elastic / elastic/crawler

Add tool to re-attempt failed bulk index payloads

Open
#66 3 comments 0 reactions 0 assignees View on GitHub
complexity:medium enhancement priority:low
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

### Problem Description

When crawler fails all retry attempts when bulk indexing a payload into Elasticsearch, it will save the payload as a file to the directory `output/failed_payloads//`. This file's existence is logged to the user but no instructions are provided on what to do with it, and no tools exist currently to easily re-index this data.

### Proposed Solution

Two things should be added to address this

- [ ] The file's payload is already perfect for a bulk index operation. The CLI should have a "re-attempt indexing of failed payloads" command that will attempt to index these files.
- [ ] Documentation should be added explaining what to do with these files.

### Alternatives

Some documentation on how to do this manually might be enough, but it would require users to use Elasticsearch API in the shell which might be a barrier for some.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.