airbytehq / airbytehq/airbyte

source-s3: CSV newlines_in_values option silently dropped in v4 migration

Aperta
#76,853 2 commenti 1 reazione 0 assegnatari Vedi su GitHub
area/connectors autoteam bug connectors/source/s3 team/extensibility
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

## Summary

The `newlines_in_values` CSV option from source-s3 v3 was silently dropped in the v4 migration. Users who relied on this setting to handle embedded newlines in CSV values now experience sync failures when upgrading to v4, with no migration path or documentation about the change.

## What changed (v3 → v4)

**v3 (≤ 3.x)** used PyArrow to parse CSVs and exposed a user-facing toggle `newlines_in_values` (default: `False`). When set to `True`, PyArrow's C-level parser allowed newline characters inside quoted CSV values.

- Defined in: `airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/csv_spec.py` (lines 54-59)

**v4 (≥ 4.0.0)** switched to the file-based CDK, which uses Python's `csv.DictReader` instead of PyArrow. The new `CsvFormat` model in the CDK (`airbyte_cdk/sources/file_based/config/csv_format.py`) has no `newlines_in_values` field.

## The legacy config migration drops the field

The `LegacyConfigTransformer` (`airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py`, lines 70-143) converts v3 configs to v4 but does not map `newlines_in_values` at all. This is even asserted in unit tests — the test input has `newlines_in_values: True`, and the expected output strips it (`unit_tests/v4/test_legacy_config_transformer.py`, lines 115-148).

The v4 migration guide (`docs/integrations/sources/s3-migrations.md`) does not mention this change.

## Practical effect

1. **The user-facing knob is gone.** A v3 user who had `newlines_in_values: True` has no equivalent v4 setting. Behavior is now hard-coded to whatever Python's `csv.DictReader` does.

2. **v4's parser is stricter about malformed rows.** When a multi-line value isn't fully RFC-4180-quoted, or when field counts don't match the header due to a stray unquoted newline, the CDK raises `ERROR_PARSING_RECORD_MISMATCHED_COLUMNS` / `ERROR_PARSING_RECORD_MISMATCHED_ROWS` (see `airbyte_cdk/sources/file_based/file_types/csv_parser.py`, lines 84-113). In v3 with `newlines_in_values: True`, PyArrow was more permissive about these edge cases.

3. **Potential `newline=""` issue.** v4's `stream_reader.open_file` opens files via `smart_open.open(..., mode="r", encoding=...)` without `newline=""`. Per Python's csv docs, this can mis-handle embedded newlines when line endings are translated — so CRLF vs LF vs lone-CR files can behave differently than the PyArrow implementation did.

## Current workaround

- Use `ignore_errors_on_fields_mismatch: true` in the v4 stream's CSV format config to skip offending rows instead of failing the sync.
- Ensure fields with embedded newlines are fully double-quoted per RFC 4180, and confirm the file uses LF or standard CRLF (not lone CR).
- If the file is not RFC 4180-compliant, pre-process before S3 ingest.

## Suggested fix

One or more of:
1. **Re-add a `newlines_in_values` option** (or equivalent) in the file-based CDK's `CsvFormat` that opens the file with `newline=""` to let `csv.DictReader` handle embedded newlines correctly per Python docs.
2. **Update the legacy config migration** (`legacy_config_transformer.py`) to translate `newlines_in_values` into whatever the v4 equivalent is.
3. **Document the behavioral change** in `docs/integrations/sources/s3-migrations.md` and mention the `ignore_errors_on_fields_mismatch` workaround.

## Reported by

Community user via Slack.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12046

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.