opensearch-project / opensearch-project/data-prepper

[BUG] Date processor does not always include timezone

Open
#5,600 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug

The date processor does not always produce the correct format string as output. I can see this happens when the date is from a previous year. But, I have not yet looked to see if other situations could cause this.

To Reproduce

  1. Create a pipeline with a date processor similar to the following:
    - date:
        match:
          - key: "calculated_at"
            patterns: [ "YYYY-MM-dd" ]
        destination: "calculated_at"
        output_format: "yyyy-MM-ddXXX"
        source_timezone: "UTC"
        destination_timezone: "UTC"
        locale: "en_US"
  1. Run the pipeline.

  2. Ingest data with the current year and last year:

curl http://localhost:2021/log/ingest -X POST -H 'Content-Type: application/json' -d '[{"calculated_at": "2024-12-31"},{"calculated_at": "2025-01-01"}]'

See the incorrect results:

data-prepper           | {"id":"year2024","calculated_at":"2024-12-31","calculated_at_text":"2024-12-31"}
data-prepper           | {"id":"year2025","calculated_at":"2025-01-01Z","calculated_at_text":"2025-01-01Z"}

Expected behavior

The date from 2024 should include the Z in the format.

Screenshots

N/A

Environment (please complete the following information):

Data Prepper 2.10.2

Additional context

This also prevents ingestion when using an upsert and a document Id. Interestingly, OpenSearch will accept the data when using insert.

Full pipeline sample:

date-test:
  delay: 10
  source:
    http:

  processor:
    # curl http://localhost:2021/log/ingest -X POST -H 'Content-Type: application/json' -d '[{"calculated_at": "2024-12-31"},{"calculated_at": "2025-01-01"}]'
    - date:
        match:
          - key: "calculated_at"
            patterns: [ "YYYY-MM-dd" ]
        destination: "calculated_at"
        output_format: "yyyy-MM-ddXXX"
        source_timezone: "UTC"
        destination_timezone: "UTC"
        locale: "en_US"
#    - add_entries:
#        entries:
#          - key: calculated_at
#            format: '${calculated_at}Z'
#            overwrite_if_key_exists: true
#            add_when: length(/calculated_at) == 10
    - copy_values:
        entries:
          - from_key: calculated_at
            to_key: calculated_at_text

  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: myStrongPassword123!
        document_id: "${/id}"
        action: upsert
        index: test_date_with_timezone
        flush_timeout: -1
        template_type: index-template
        template_content: >
          {
            "template" : {
              "mappings" : {
                "properties" : {
                  "calculated_at" : {
                    "type" : "date",
                    "format": "yyyy-MM-ddXXX"
                  },
                  "calculated_at_text" : {
                    "type" : "text"
                  }
                }
              }
            }
          }
    - stdout:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the date processor implementation and its tests, then reproduce the pipeline with the two dates shown in the issue. The fix is done when both outputs include the UTC Z suffix and the documented upsert ingestion succeeds without the timezone-related format mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.