"corrupt deflate stream" when inferring schema from CSV file during CREATE EXTERNAL TABLE
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Describe the bug**
Inferring the schema does not work with a compressed file, in some cases.
**To Reproduce**
Create a file like this one, and gzip it:
```
❯ cat file1.csv
Region,Units
United States,10
Canada,20
United States,10
Canada,20
United States,10
Canada,20
Canada,20
❯ gzip file1.csv
```
I couldn't duplicate this error with a gzipped file of exact repeating rows. It requires some variation in the rows (or there's something else that triggers it).
Then attempt to make a table from it:
```
❯ create external table file_csv stored as csv with header row compression type gzip location './file1.csv.gz';
ArrowError(CsvError("corrupt deflate stream"))
```
**Expected behavior**
It shouldn't throw an error and should be able to create the external table.
**Additional context**
It works fine with the same file if you specify the table schema:
```
❯ create external table file_csv(a string,b int) stored as csv with header row compression type gzip location './file1.c
sv.gz';
0 rows in set. Query took 0.000 seconds.
❯ select * from file_csv;
+---------------+----+
| a | b |
+---------------+----+
| United States | 10 |
| Canada | 20 |
| United States | 10 |
| Canada | 20 |
| United States | 10 |
| Canada | 20 |
| Canada | 20 |
+---------------+----+
7 rows in set. Query took 0.003 seconds.
```
Contributor guide
Research direction
Start by reproducing the CREATE EXTERNAL TABLE command against file1.csv.gz, then trace the CSV schema-inference path used when compression type gzip is specified. Compare it with the explicit-schema command, and consider the issue done when schema inference succeeds and SELECT returns all seven rows without a corrupt deflate stream error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100