apache / apache/datafusion

Aggregations on partition columns may fail if `wrap_partition_value_in_dict` is used.

Open
#16,550 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I am using `delta-rs` and `datafusion` to query my Delta table. Relevant sections of the code I found useful when debugging this:

https://github.com/delta-io/delta-rs/blob/2cc8081caf89ec4aa162d8c9a903d3b9890fcac9/crates/core/src/delta_datafusion/mod.rs#L679
https://github.com/apache/datafusion/blob/20a723b7b6d91da57fe6abea8ecac08ea5267a89/datafusion/datasource/src/file_scan_config.rs#L1527
https://github.com/apache/datafusion/blob/20a723b7b6d91da57fe6abea8ecac08ea5267a89/datafusion/common/src/scalar/mod.rs#L2108

So if I have a query like,
```sql
SELECT
"SpanId" as spanId,
"TraceId" as traceId,
-- TracingProjectId is a partition column
FIRST_VALUE("TracingProjectId") as tracingProjectId
FROM tbl
WHERE "TfyTenantId" = 'truefoundry'
AND "TraceId" = 'fb09d0c9b49136bb161464b3e32c5083'
AND "ParentSpanId" = '662b7388122dfb79'
AND ("MlRepoId", "TracingProjectId") IN ((1089, '222fde49-1f7a-4752-8ec1-06bcdbf570c5'), (1234, '2a68e4f6-eda0-4b50-aeb1-aee55a8c52df'))
AND "Timestamp" >= to_timestamp('2025-04-15 07:30:00')
GROUP BY "TraceId", "SpanId"
LIMIT 10
OFFSET 0
```

I get,
```
Internal error: Can not create index of type UInt16 from value 65536.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```

If I change `FIRST_VALUE("TracingProjectId") as tracingProjectId` to `FIRST_VALUE(CAST("TracingProjectId" AS VARCHAR)) as tracingProjectId`, it works.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the aggregation query using the reported partition-column setup, then inspect delta_datafusion/mod.rs around line 679 and DataFusion's file_scan_config.rs around line 1527 and scalar/mod.rs around line 2108. Compare the failing FIRST_VALUE call with the VARCHAR cast workaround; done means the uncast partition-column aggregation no longer produces the UInt16 index error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.