elastic / elastic/curator

Curator reindex KeyError randomly

Open
#1,502 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.1k
Forks
627
PR merge metrics
No merged PRs in 30d

Description

## To submit a bug or report an issue
When doing reindex, occasionally in 5-7% of cases I get
```Failed to complete action: reindex. : 'indices'```

### Expected Behavior
The task I have is quite simple - I get all index names that are readonly, and then one by one for each index:
- I create I new index with needed settings;
- I start reindex task;
- I delete source index;
I do it with AWS Elasticsearch. It works as it should, but sometimes fails.

### Actual Behavior
The task runs fine, but sometimes (I could only connect it with big number of running reindex tasks) I get:
```
2020-01-15 11:11:27,316 INFO Trying Action ID: 18, "reindex": Reindex ark-r2-node-2020.01.14
2020-01-15 11:11:29,081 ERROR Failed to complete action: reindex. : 'indices'
```
As I have a lot of reindex tasks - I get such error 5-7 times out of 100 (and this is oncce in an hour). There's no pattern, it could be any index at any time. The index could be 30Mb, could be 40Gb. It always reindexes from 3 primary 1 replica to 1 primary 1 replica index.

### Specifications
AWS ES 7.1
Curator 5.8.1

## Context (Environment)
Can we somehow catch this error and do retry? Maybe it is happening because index for reindex is in process of creation, and we can add delay option? Or maybe there's even a way to catch the verbose output to define the reason?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.