duckdb / duckdb/duckdb

DuckDB fails on parquet with invalid UTF8 error that passed clickhouse and starrocks fine

Open
#20,906 4 comments 1 reaction 0 assignees View on GitHub
under review
Dominant language
C++
Stars
41.2k
Forks
3.8k
Avg merge
2d 7h
Merged PRs (30d)
515

Description

### What happens?

THis error:

```
An execution error has occurred: Invalid Input Error: Invalid string encoding found in Parquet file: value "src_v1_upp_ahttps://aristatsis.gr/2025/08/27/\xCF\x80\xCF\x81\xCE\xBF\xCF\x83\xCF\x89\xCF\x80\xCE\xB9\xCE\xBA\xCE\xBF\xCF" is not valid UTF8!
[2026-02-11 00:49:08] UNKNOWN: An execution error has occurred: Invalid Input Error: Invalid string encoding found in Parquet file: value "src_v1_upp_ahttps://aristatsis.gr/2025/08/27/\xCF\x80\xCF\x81\xCE\xBF\xCF\x83\xCF\x89\xCF\x80\xCE\xB9\xCE\xBA\xCE\xBF\xCF" is not valid UTF8!
```

ok, so this came from some random parquet (which of my 3100 files of 3.3TB was it? error doesn't say.

The data came from clickhouse via parquet to starrocks. Then starrocks through mv's back to starrocks. Then starrocks to parquet. And now DuckDB is the only one that won't read it. I cannot find nor fix it easily.

Polars has `encoding="utf8-lossy"` flag for its file formats allowing you to skip over these type of issues (or it truncates at error, I don't care which). Also if we have to find such errors being able to treat a column as binary and ask "valid_utf8_encoding" to find problems and see if we want to live with the count of them or not. I have a handful of these in 30 billion records now, and am stuck without going up and regenerating all parquets and trying to detect in upstream system.

Can we have a similar flag? Having something slip into a large set of data in the middle of processing partitions and having an unrecoverable problem in an unknown location is problematic. I'd rather lose the value in this field than to have my entire process stop with no easy resolution but to export all files again and try to wrap them with something specific to duckdb's handling.

Also, PLEASE report the file with the error such as this.

### To Reproduce

The error message indicates the content of the field.

The error message does not give me enough info to find this silly file.

### OS:

amd64

### DuckDB Version:

1.4.4

### DuckDB Client:

duckdb

### Hardware:

_No response_

### Full Name:

Jayson Minard

### Affiliation:

locals.com

### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

- [x] Yes, I have

### Did you include all code required to reproduce the issue?

- [ ] Yes, I have

### Did you include all relevant data sets for reproducing the issue?

No - I cannot easily share my data sets due to their large size

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or reproducible dataset are named. Start by examining DuckDB's Parquet reader and its UTF-8 validation error path, then determine how an optional lossy-encoding mode and the offending file path could be specified and tested. Done means the requested behavior is defined and covered for invalid UTF-8 input.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.