redpanda-data / redpanda-data/connect
CSV parsing skips first column
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Hi
I have a simple CSV:
Date;Country;City;Start HRFull;Total Users;Conference Id
2023/08/05;Czech Republic;;00:08:59;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;00:33:01;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;06:58:24;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;06:58:47;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;06:59:51;1;e07f5389-6f2b-46fc-a3a6-f1627169d2fc
2023/08/05;Czech Republic;;08:01:52;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;08:20:00;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
2023/08/05;Czech Republic;;08:32:53;1;2ea6df2c-5378-4669-85f7-0c46beb42bbc
2023/08/05;Czech Republic;;08:39:08;1;7f6de524-1edd-4a2f-a96b-36b4afc68705
and a simple pipeline:
input:
file:
paths:
- test.csv
codec: csv:;
pipeline:
processors:
- mapping: |
root = [
this.Date,
this.Country,
this.City
]
in the output the first element (this.Date) is null:
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
[null,"Czech Republic",""]
but if I remove the pipeline the output is:
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"00:08:59","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"00:33:01","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:58:24","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:58:47","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"e07f5389-6f2b-46fc-a3a6-f1627169d2fc","Country":"Czech Republic","Start HRFull":"06:59:51","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:01:52","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:20:00","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"2ea6df2c-5378-4669-85f7-0c46beb42bbc","Country":"Czech Republic","Start HRFull":"08:32:53","Total Users":"1","Date":"2023/08/05"}
{"City":"","Conference Id":"7f6de524-1edd-4a2f-a96b-36b4afc68705","Country":"Czech Republic","Start HRFull":"08:39:08","Total Users":"1","Date":"2023/08/05"}
so "Date" is parsed correctly from CSV
If I add a dummy first column this.Date get the right value.
It does not work on macOS, I tested on Linux and it works as expected.
Issue is related to line endings. The CSV was exported from Excel, rewriting it worked. Probably CSV parser should handle every line ending combination.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CSV codec and its handling of macOS and Excel line endings. Reproduce the semicolon-delimited input with the mapping pipeline and compare behavior across line-ending combinations. Done means the first Date field is preserved consistently, while existing CSV parsing remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100