apache / apache/datafusion

Substrait + identifier normalization flags issues

Open
#14,832 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

This is a report from @lmwnshn and is part of
- https://github.com/apache/datafusion/issues/14373

enable_ident_normalization: I think there may be some extra complications in my situation arising from the use of Substrait (or perhaps Substrait table names are treated as quoted?

In particular, with `enable_ident_normalization=false`, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:

### To Reproduce

Please see
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L37

In particular, with enable_ident_normalization=false, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L12-L19

So I had to hack the parquet files up a bit
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/fix_parquet.py#L11-L23

and switch to uppercased column names + register tables as uppercase
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion.py#L12-L19
to get the Substrait plan to execute successfully.

### Expected behavior

I expect that the I can register lowercase tablenames and lowercase column names in the parquet files

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction in run_datafusion_ident.py, especially the enable_ident_normalization setting and the lowercase table and column registration, then compare it with run_datafusion.py and fix_parquet.py. Trace how the Substrait plan handles those identifiers and verify that the original lowercase parquet tables and columns register and execute without modifying the files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.