apache / apache/datafusion-comet

Preserve dictionary encoding through native expressions where possible

Open
#4,228 0 comments 0 reactions 0 assignees View on GitHub
area:expressions area:scan enhancement priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet flattens Parquet dictionary-encoded string columns at the read boundary in `native/core/src/parquet/parquet_support.rs:170` (the `take(values, keys, None)` branch in `parquet_convert_array`). Because `QueryPlanSerde` serializes Spark `StringType` as Arrow `Utf8`, the requested `to_type` is never `Dictionary`, so this branch always fires. Every native expression sees a flat `Utf8` array, even when the source column has very low cardinality.

This loses the dict advantage. For low-cardinality columns (URL hosts, country codes, status flags, enum-shaped strings) an expression like `unhex`, `length`, `lower`, `upper`, `regexp_replace`, etc. evaluates per-row instead of per-unique-value, multiplying the work by `N / M` where `M` is the dictionary size.

### Describe the potential solution

Three layers, ordered by feasibility:

1. **Per-expression dict-aware paths.** Many expressions (`substring`, `rlike`, `cast`, the `temporal` kernels) already operate on dictionary values and reassemble. Audit the rest and apply the pattern from `string_funcs/substring.rs:54-60`. Document this in `adding_a_new_expression.md`.

2. **Plan-aware read boundary.** Make `parquet_convert_array`'s flatten-vs-preserve a function of whether the immediate downstream expression accepts dict input, not just the requested `to_type`. Needs a small planner pass to thread that hint back to the scan.

3. **Spark-boundary materialization stays.** Anywhere a batch leaves native execution (shuffle, fallback, scan output to JVM) flatten as today. Spark's row layout has no dict shape.

Until (2) lands, per-expression work in (1) only helps when a dict is produced mid-plan, which is rare. Both are needed for measurable wins.

### Additional context

Concrete starting point: pick one common high-cardinality-savings expression (e.g. `length` or `lower`), add the dict path, and microbenchmark on a 1M row column with ~100 unique values. If the speedup is in the 5-15% range expected for string-heavy TPC-DS-style queries, that is enough signal to justify the planner work in (2).

Came up while reviewing #4222.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.