apache / apache/datafusion

"corrupt deflate stream" when inferring schema from CSV file during CREATE EXTERNAL TABLE

Open
#5,041 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Describe the bug**
Inferring the schema does not work with a compressed file, in some cases.

**To Reproduce**
Create a file like this one, and gzip it:
```
❯ cat file1.csv
Region,Units
United States,10
Canada,20
United States,10
Canada,20
United States,10
Canada,20
Canada,20
❯ gzip file1.csv
```
I couldn't duplicate this error with a gzipped file of exact repeating rows. It requires some variation in the rows (or there's something else that triggers it).

Then attempt to make a table from it:
```
❯ create external table file_csv stored as csv with header row compression type gzip location './file1.csv.gz';
ArrowError(CsvError("corrupt deflate stream"))
```

**Expected behavior**
It shouldn't throw an error and should be able to create the external table.

**Additional context**
It works fine with the same file if you specify the table schema:
```
❯ create external table file_csv(a string,b int) stored as csv with header row compression type gzip location './file1.c
sv.gz';
0 rows in set. Query took 0.000 seconds.
❯ select * from file_csv;
+---------------+----+
| a | b |
+---------------+----+
| United States | 10 |
| Canada | 20 |
| United States | 10 |
| Canada | 20 |
| United States | 10 |
| Canada | 20 |
| Canada | 20 |
+---------------+----+
7 rows in set. Query took 0.003 seconds.
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CREATE EXTERNAL TABLE command against file1.csv.gz, then trace the CSV schema-inference path used when compression type gzip is specified. Compare it with the explicit-schema command, and consider the issue done when schema inference succeeds and SELECT returns all seven rows without a corrupt deflate stream error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.