apache / apache/datafusion

`schema_force_view_type` configuration not working for `CREATE EXTERNAL TABLE`

Open
#14,909 4 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`schema_force_view_type` configuration said:
```
datafusion.execution.parquet.schema_force_view_types | true | (reading) If true, parquet reader will read columns of Utf8/Utf8Large with Utf8View, and Binary/BinaryLarge with BinaryView.
```
Reference: https://datafusion.apache.org/user-guide/configs.html

I tried the following script within `datafusion-cli`
```sql
set datafusion.execution.parquet.schema_force_view_types = true;

CREATE EXTERNAL TABLE IF NOT EXISTS lineitem (
l_orderkey BIGINT,
l_partkey BIGINT,
l_suppkey BIGINT,
l_linenumber INTEGER,
l_quantity DECIMAL(15, 2),
l_extendedprice DECIMAL(15, 2),
l_discount DECIMAL(15, 2),
l_tax DECIMAL(15, 2),
l_returnflag VARCHAR,
l_linestatus VARCHAR,
l_shipdate DATE,
l_commitdate DATE,
l_receiptdate DATE,
l_shipinstruct VARCHAR,
l_shipmode VARCHAR,
l_comment VARCHAR
) STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf10/lineitem';

select arrow_typeof(l_comment) from lineitem limit 1;
```

The data type is `Utf8` instead `Utf8View`
```
+----------------------------------+
| arrow_typeof(lineitem.l_comment) |
+----------------------------------+
| Utf8 |
+----------------------------------+
```

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.