opensearch-project / opensearch-project/data-prepper

[BUG] S3 source csv codec parsing

Open
#2,471 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
S3 source with csv codec is not paring the csv file accurately. It appends an additional space before the first key.

To Reproduce
Steps to reproduce the behavior:

  1. create pipeline with S3 source and csv codec
  2. ingest some csv data

Pipeline config:

log-pipeline:
  source:
    s3:
      notification_type: "sqs"
      compression: none
      codec:
        csv:
      sqs:
        queue_url: "https://sqs.us-east-1.amazonaws.com/<account-id>/<queue-name>"
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::<account-id>:role/<role>"
      on_error: "retain_messages"

  processor:
    - date:
        from_time_received: true
        destination: "@timestamp"
   - delete_entries:
       with_keys: ["col1"]

  sink:
    - stdout:

Input:

col1,col2
a,101
b,102

Output:

{" col1":"a","col2":"101","s3":{"bucket":"bucket-name","key":"Book1.csv"},"@timestamp":"2023-04-10T14:48:53.411-05:00"}
{" col1":"b","col2":"102","s3":{"bucket":"bucket-name","key":"Book1.csv"},"@timestamp":"2023-04-10T14:48:53.580-05:00"}

Expected behavior

  • There shouldn't be a space before col1 in above output and delete_entries should delete the key.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the S3 source with the CSV codec using the pipeline configuration and input shown in the issue. Trace the S3 source and CSV codec handling to determine why the first key includes an extra space or character, then verify that the expected keys are parsed and delete_entries removes col1.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.