Processing s3 source csv with newlines

Describe the bug

In my setup I receive CDC records from AWS DMS that get put in S3. Those records contain data from the DB that might contain a field that has newlines in it

An example of a possible CSV file is:

D,metadata,public,Lyrics,"I don’t love you like I love my dog

I don’t love you like I love my dog

My dog wanders with me all day

My dog wanders with me all day



People leave you and let you down

People leave you and let you down

But my dog will always stay

Just a footstep away

People leave you and they let you down



When my dog goes I will go too

When my dog goes I will go too

Over that bridge into the blue

I’m not going anywhere with you

Cause when my dog goes I will go too



I don’t love you like I love my dog

I don’t love you like I love my dog",122474,exiftool,2338974,,,
U,track,public,122474,dog,dog,dog,2025-03-04 10:11:49.924523,0,80370,,web_upload:///dog.mp3,,false,,6,2025-03-03 16:48:30.271660,31274,audio,true,155,internal,,false,0,2025-03-03 16:48:34.372470,1,someurl,false,10,,,,1100,,false,,true,31942,,,true,,,,,model123,,,,,,,,,,

In this case the lyrics have multiple lines.
Due to the nature of CDC updates, each line will have a different number of columns which causes the S3 source CSV codec to crash and suggest to use the csv processor.

Unfortunately this means setting the codec to newline which means Data prepper will see the lyrics newlines as separate rows meaning incorrect events will be sent to my pipeline crashing it (

simple-sample-pipeline:
  workers: 1
  source:
    s3:
      acknowledgments: true
      notification_type: "sqs"
      compression: "none"
      codec:
        newline:
      sqs:
        queue_url: ""
        maximum_messages: 10
        visibility_timeout: "30s"
        visibility_duplication_protection: true
      aws:
        region: ""
        sts_role_arn: ""
  processor:
    - csv:
        column_names: ["op", "table", "schema"]
      # Here I would do Lambda processing of each record but will receive incorrect data
  sink:
      - stdout
      # ...

To Reproduce
Steps to reproduce the behavior:

1. Create a CSV file on S3 with the text above
2. Trigger data prepper and look at it printing incorrect records

Expected behavior

Ability to handle cases like this. Probably means S3 codec is able to handle dynamic columns so we don't need to split the file?

Environment (please complete the following information):

• OS: [e.g. Ubuntu 20.04 LTS] Ubuntu 22.04.3 LTS
• Version [e.g. 22] Latest (opensearchproject/data-prepper:latest)

Additional context

I should be able to fix it by having DMS only expose some columns this is I am not sure if should be seen as a bug or a feature request for the S3 source csv to handle dynamic columns so that I wouldn't have to do this?

Start at the S3 source CSV codec and reproduce the issue with the multiline sample CSV from this report. Trace how quoted fields containing newlines are split and compare that with the newline codec behavior. Done means the multiline lyrics remain part of one CSV record and the resulting events are not incorrectly split.