opensearch-project / opensearch-project/data-prepper
[BUG] Date processor does not always include timezone
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The date processor does not always produce the correct format string as output. I can see this happens when the date is from a previous year. But, I have not yet looked to see if other situations could cause this.
To Reproduce
- Create a pipeline with a date processor similar to the following:
- date:
match:
- key: "calculated_at"
patterns: [ "YYYY-MM-dd" ]
destination: "calculated_at"
output_format: "yyyy-MM-ddXXX"
source_timezone: "UTC"
destination_timezone: "UTC"
locale: "en_US"
-
Run the pipeline.
-
Ingest data with the current year and last year:
curl http://localhost:2021/log/ingest -X POST -H 'Content-Type: application/json' -d '[{"calculated_at": "2024-12-31"},{"calculated_at": "2025-01-01"}]'
See the incorrect results:
data-prepper | {"id":"year2024","calculated_at":"2024-12-31","calculated_at_text":"2024-12-31"}
data-prepper | {"id":"year2025","calculated_at":"2025-01-01Z","calculated_at_text":"2025-01-01Z"}
Expected behavior
The date from 2024 should include the Z in the format.
Screenshots
N/A
Environment (please complete the following information):
Data Prepper 2.10.2
Additional context
This also prevents ingestion when using an upsert and a document Id. Interestingly, OpenSearch will accept the data when using insert.
Full pipeline sample:
date-test:
delay: 10
source:
http:
processor:
# curl http://localhost:2021/log/ingest -X POST -H 'Content-Type: application/json' -d '[{"calculated_at": "2024-12-31"},{"calculated_at": "2025-01-01"}]'
- date:
match:
- key: "calculated_at"
patterns: [ "YYYY-MM-dd" ]
destination: "calculated_at"
output_format: "yyyy-MM-ddXXX"
source_timezone: "UTC"
destination_timezone: "UTC"
locale: "en_US"
# - add_entries:
# entries:
# - key: calculated_at
# format: '${calculated_at}Z'
# overwrite_if_key_exists: true
# add_when: length(/calculated_at) == 10
- copy_values:
entries:
- from_key: calculated_at
to_key: calculated_at_text
sink:
- opensearch:
hosts: [ "https://opensearch:9200" ]
insecure: true
username: admin
password: myStrongPassword123!
document_id: "${/id}"
action: upsert
index: test_date_with_timezone
flush_timeout: -1
template_type: index-template
template_content: >
{
"template" : {
"mappings" : {
"properties" : {
"calculated_at" : {
"type" : "date",
"format": "yyyy-MM-ddXXX"
},
"calculated_at_text" : {
"type" : "text"
}
}
}
}
}
- stdout:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the date processor implementation and its tests, then reproduce the pipeline with the two dates shown in the issue. The fix is done when both outputs include the UTC Z suffix and the documented upsert ingestion succeeds without the timezone-related format mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100