redpanda-data / redpanda-data/connect

CSV parsing skips first column

Open
#2,117 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs more info
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Hi
I have a simple CSV:

Date;Country;City;Start HRFull;Total Users;Conference Id
2023/08/05;Czech Republic;;00:08:59;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;00:33:01;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;06:58:24;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;06:58:47;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;06:59:51;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;08:01:52;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;08:20:00;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;08:32:53;1;2ea6df2c-5378-4669-85f7-0c46beb42bbc
2023/08/05;Czech Republic;;08:39:08;1;7f6de524-1edd-4a2f-a96b-36b4afc68705

and a simple pipeline:

input:
  file:
    paths:
    - test.csv
    codec: csv:;

pipeline:
  processors:
  - mapping: |
      root = [
        this.Date,
        this.Country,
        this.City
      ]

in the output the first element (this.Date) is null:

[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]

but if I remove the pipeline the output is:

{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"00:08:59","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"00:33:01","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:58:24","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:58:47","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:59:51","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:01:52","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:20:00","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"2ea6df2c-5378-4669-85f7-0c46beb42bbc","Country":"Czech Republic","Start HRFull":"08:32:53","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:39:08","Total Users":"1","Date":"2023/08/05"}

so "Date" is parsed correctly from CSV

If I add a dummy first column this.Date get the right value.

It does not work on macOS, I tested on Linux and it works as expected.

Issue is related to line endings. The CSV was exported from Excel, rewriting it worked. Probably CSV parser should handle every line ending combination.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the CSV codec and its handling of macOS and Excel line endings. Reproduce the semicolon-delimited input with the mapping pipeline and compare behavior across line-ending combinations. Done means the first Date field is preserved consistently, while existing CSV parsing remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.