[Elasticsearch Input Plugin] tracking_field placeholder :last_value not replaced when connected to Elasticsearch 7.17.29
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
1. Logstash version: 8.19.13
2. Elasticsearch version: 7.17.29
**Steps to reproduce**:
1. Elasticsearch version: 7.17.29 (source cluster)
2. Target Elasticsearch version: 7.17.29 (destination cluster)
3. Logstash configuration:
**Steps to reproduce**:
1. Elasticsearch version: 7.17.29 (source cluster)
2. Target Elasticsearch version: 7.17.29 (destination cluster)
3. Logstash configuration:
```ruby
input {
elasticsearch {
id => "es-input-incremental"
hosts => "http://elasticsearch-qa:9200"
index => "test_index_20260403"
query => '{
"query": {
"bool": {
"must": [
{
"range": {
"insert_timestamp": {
"gt": ":last_value"
}
}
}
],
"should": [
{
"bool": {
"must_not": [
{
"exists": {
"field": "source_es"
}
}
]
}
},
{
"bool": {
"must_not": [
{
"term": {
"source_es": "es1"
}
}
]
}
}
],
"minimum_should_match": 1
}
},
"sort": [
{
"insert_timestamp": {
"order": "asc",
"numeric_type": "date_nanos"
}
}
]
}'
tracking_field => "insert_timestamp"
tracking_field_seed => "1970-01-01T00:00:00.000Z"
schedule => "*/5 * * * *"
schedule_overlap => false
last_run_metadata_path => "/usr/share/logstash/workspace/data/last_run_es_input"
size => 1000
scroll => "5m"
slices => 1
docinfo => true
docinfo_target => "[@metadata][es]"
docinfo_fields => ["_index", "_id"]
}
}
```
4. Logstash log output
```ruby
[ERROR][l.i.e.scroll] Tried scroll paginated search unsuccessfully {:message=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [:last_value] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [:last_value] with format [strict_date_optional_time||epoch_millis]]\"}],...}"}
```
5. index mapping
```ruby
{
"dynamic_templates": [
{
"dataFormat": {
"mapping": {
"type": "date"
},
"match": "*Date"
}
},
{
"timeFormat": {
"mapping": {
"type": "date"
},
"match": "*Time"
}
},
{
"noAnalyzed": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"trackAngle": {
"type": "float"
},
"insert_timestamp": {
"type": "date"
},
"latitude": {
"type": "float"
},
"uniqueProductIdentificationCode": {
"type": "keyword"
},
"_class": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"operatingStatus": {
"type": "long"
},
"realNameRegistrationSign": {
"type": "keyword"
},
"relativeAltitude": {
"type": "float"
},
"speed": {
"type": "float"
},
"longitude": {
"type": "float"
},
"timestamp": {
"type": "long"
}
}
}
```
6. doc
```ruby
{
"_index": "uav_runtime_data_20260403",
"_type": "_doc",
"_id": "KqulZp0BXFn5C3AdxHAu",
"_score": null,
"_source": {
"insert_timestamp": "2026-04-07T06:33:52.941754653Z",
"latitude": 39.960405748599435,
"speed": 29.707272993776467,
"relativeAltitude": 117.95207404758052,
"trackAngle": 155.39948491936858,
"uniqueProductIdentificationCode": "testUav_1775149200002",
"_class": "team.abc.laatc.base.model.UavRuntimeDataModel",
"id": "testUav_17751492000021775149200001",
"operatingStatus": 2,
"realNameRegistrationSign": "TEST_REG_2",
"longitude": 116.48075115518893,
"timestamp": 1775149200002
},
"sort": [
1775543632941
]
}
```
Contributor guide
Research direction
Start at the Elasticsearch input plugin configuration, focusing on tracking_field, tracking_field_seed, last_run_metadata_path, and query handling. Reproduce the scheduled input against Elasticsearch 7.17.29 and trace where :last_value should be substituted; done means the generated query uses the stored value and the scroll search succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, ruby
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100