elastic / elastic/curator

Log the index name when a field_stat filter query raises an exception

Open
#1,265 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.1k
Forks
627
PR merge metrics
No merged PRs in 30d

Description

Note that this is a resubmission of #1200 except this time the problem is not resolved by ignoring empty indices (#1167).

### Expected Behavior
Sometimes, due to misconfiguration or bad data in Elasticsearch, the "age" filtertype with "field_stats" source may raise an exception. In my case, the exception was caused by an index with documents, but **none containing the timestamp field** specified in the filter. This results in the field_stats aggregation returning no min/max values, causing an exception. When this happens, the system should log the name of the index which resulted in the failed aggregation query so that the user can troubleshoot the issue.

### Actual Behavior
The system logs the exception, but not the index which was being processed at the time:
```
13:33:03,954 DEBUG curator.indexlist _get_field_stats_dates:263 RESPONSE: {'took': 7, 'timed_out': False, '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0}, 'hits': {'total': 0, 'max_score': 0.0, 'hits': []}, 'aggregations': {'min': {'value': None}, 'max': {'value': None}}}
13:33:03,954 DEBUG curator.indexlist _get_field_stats_dates:267 r: {'min': {'value': None}, 'max': {'value': None}}
13:33:03,954 ERROR curator.cli run:184 Failed to complete action: close. : int() argument must be a string, a bytes-like object or a number, not 'NoneType'
```

### Steps to Reproduce the Problem
1. Create an Elasticsearch index named `my-empty-index` with one or more documents lacking the "@timestamp" field
2. Configure a Curator action using an age/field_stats filter like:
```yaml
actions:
1:
action: delete_indices
description: "Delete old indices"
options:
timeout_override: 300
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: my-empty-index
- filtertype: age
direction: older
source: field_stats
field: '@timestamp'
stats_result: max_value
unit: days
unit_count: 2
```
3. Run the curator action

### Specifications

- Version: 5.5.4
- Platform: Mac OSX
- Subsystem: ???

## Context (Environment)
I'm using a combination of pattern and field_stats age filters to select indices for closing/deletion in a production ES cluster. A new index was created which matched the pattern but did not contain the timestamp field. The job started failing, but it was difficult to track down which index was causing the issues out of >100 which match the pattern.

## Detailed Description
I propose, at a minimum, that the system should log the name of the index being processed when this type of error occurs. Even better, it would be nice to have a filter option to ignore indexes that fail the aggregation query for one reason or another.

Contributor guide

Open the contributing guide

Research direction

Start at curator.indexlist._get_field_stats_dates and reproduce the field_stats age-filter failure with an index whose documents lack the timestamp field. The issue is done when the resulting exception log identifies the index being processed, without changing the requested behavior of the filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
databases, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.