Substrait + identifier normalization flags issues
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
This is a report from @lmwnshn and is part of
- https://github.com/apache/datafusion/issues/14373
enable_ident_normalization: I think there may be some extra complications in my situation arising from the use of Substrait (or perhaps Substrait table names are treated as quoted?
In particular, with `enable_ident_normalization=false`, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:
### To Reproduce
Please see
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L37
In particular, with enable_ident_normalization=false, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L12-L19
So I had to hack the parquet files up a bit
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/fix_parquet.py#L11-L23
and switch to uppercased column names + register tables as uppercase
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion.py#L12-L19
to get the Substrait plan to execute successfully.
### Expected behavior
I expect that the I can register lowercase tablenames and lowercase column names in the parquet files
### Additional context
_No response_
Contributor guide
Research direction
Start with the reproduction in run_datafusion_ident.py, especially the enable_ident_normalization setting and the lowercase table and column registration, then compare it with run_datafusion.py and fix_parquet.py. Trace how the Substrait plan handles those identifiers and verify that the original lowercase parquet tables and columns register and execute without modifying the files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100