opensearch-project / opensearch-project/data-prepper
[BUG] S3 source csv codec parsing
Open
Nobody has claimed this yet.
bug
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
S3 source with csv codec is not paring the csv file accurately. It appends an additional space before the first key.
To Reproduce
Steps to reproduce the behavior:
- create pipeline with S3 source and csv codec
- ingest some csv data
Pipeline config:
log-pipeline:
source:
s3:
notification_type: "sqs"
compression: none
codec:
csv:
sqs:
queue_url: "https://sqs.us-east-1.amazonaws.com/<account-id>/<queue-name>"
aws:
region: "us-east-1"
sts_role_arn: "arn:aws:iam::<account-id>:role/<role>"
on_error: "retain_messages"
processor:
- date:
from_time_received: true
destination: "@timestamp"
- delete_entries:
with_keys: ["col1"]
sink:
- stdout:
Input:
col1,col2
a,101
b,102
Output:
{" col1":"a","col2":"101","s3":{"bucket":"bucket-name","key":"Book1.csv"},"@timestamp":"2023-04-10T14:48:53.411-05:00"}
{" col1":"b","col2":"102","s3":{"bucket":"bucket-name","key":"Book1.csv"},"@timestamp":"2023-04-10T14:48:53.580-05:00"}
Expected behavior
- There shouldn't be a space before
col1in above output anddelete_entriesshould delete the key.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the S3 source with the CSV codec using the pipeline configuration and input shown in the issue. Trace the S3 source and CSV codec handling to determine why the first key includes an extra space or character, then verify that the expected keys are parsed and delete_entries removes col1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100