apache / apache/datafusion

Enable placement on more `scalar_fn`s to allow lowering and pushdown to `TableScan`s

Open
#25,036 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

We should use the new placement API for more scalar fns. This allows better query plans for example in vortex we get a large speed up for clickbench query q29 (below) when pushing down `OCTET_LENGTH("URL")` since we never need load the URL str bytes

```sql
SELECT "CounterID", AVG(OCTET_LENGTH("URL")) AS l, COUNT(*) AS c FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; -- ClickHouse's length counts bytes, not code points as standard SQL.
```

Scalar fn:

- [ ] bit/octet/''-length
- [ ] todo

Please add any more and we can update the list

Contributor guide

Open the contributing guide

Research direction

Start with the placement API and trace the OCTET_LENGTH example through lowering to TableScan, using ClickBench query q29 as the concrete case. Identify additional scalar functions beyond the listed bit/octet/''-length items, add placement support for the selected functions, and confirm that eligible expressions can be pushed into the resulting plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.