apache / apache/texera

CSV/JSONL scan sources silently drop rows that fail type parsing

Open
#6,279 1 comment 0 reactions 1 assignee Claimed by @eugenegujing View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

The CSV and JSONL scan source operators infer each column's type from only the first `INFER_READ_LIMIT` (= 100) rows. When a **later** row holds a value that does not parse as the inferred type, the operator **silently discards the entire row** — no warning, no skipped-row count, no console message, no log entry. The user has no way to learn that data was lost, so every downstream count, aggregate, join, and model is computed on a silently truncated dataset.

Cells that are simply **empty** are kept as `null` (the row survives), while cells that **fail to parse** cause the **whole row** to vanish. The operator treats "missing" and "malformed" differently, and the more destructive path is the silent one. Rejecting malformed input is defensible but doing it **silently** would be a bug

**What I expected to happen:** the loss should not be silent. At minimum the number of skipped rows should be surfaced (execution statistic / console warning), or the scan should fail loudly and name the offending row and column. Losing data with zero signal is the defect.

### How to reproduce?

Using the `diabetes` dataset (400 data rows). Type inference only samples the first 100 rows, so inject a bad value **after** row 100 into a column that is all-integer in the first 100 rows (e.g. `age`), which makes inference stably INTEGER:

1. Create a corrupted copy that puts a non-integer into `age` on three rows past row 100:
```bash
# data rows 150, 250, 350 -> age becomes "55.5" (fails Integer parsing)
awk -F, 'BEGIN{OFS=","} NR==1{print; next}
{n++; if(n==150||n==250||n==350){$8="55.5"} print}' diabetes.csv > diabetes_b2.csv
```
2. Upload `diabetes_b2.csv` as a dataset and point a **CSV File Scan** operator at it.
3. Run the workflow and look at the scan operator's output row count.

**Expected:** 403 rows accounted for, or a visible signal that 3 rows were skipped.

**Actual:** the scan emits **400** rows. The 3 rows containing `55.5` are gone, with no warning, no skipped-row count, and nothing in the console — silent data loss. (JSONL, ParallelCSV, and the `csvOld` variant share the same behavior.)

Image

### Version/Branch

1.3.0-incubating-SNAPSHOT (main)

### Commit Hash (Optional)

_No response_

### What browsers are you seeing the problem on?

_No response_

### Relevant log output

```shell

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.