apache / apache/gluten

[VL] Fallback for map function

Open
#9,452 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

I see this fallback when using map function:
(25) Project: Native validation failed:
Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1381 function:validate, thrown from file:ExprCompiler.cpp line:477 function:compileRewrittenExpression, reason:Scalar function map not registered with arguments: (VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE). Found function registered with the following signatures:
((K,V) -> map(K,V))
((K,V,K,V) -> map(K,V))
((K,V,K,V,K,V) -> map(K,V))

Is it because of this signature definition in velox here https://github.com/facebookincubator/velox/blob/00d2ca375016318d5a4cc061775e462c4f4f555e/velox/functions/sparksql/Map.cpp#L117C2-L136C15? Can we just increase it to support more than 6 inputs by chanigng the value to more than 3? @PHILO-HE
```
static std::vector> signatures() {
// For the purpose of testing we introduce up to 6 inputs
// array(K), array(V) -> map(K,V)
std::vector> signatures;
constexpr int kNumberOfSignatures = 3;
signatures.reserve(kNumberOfSignatures);
for (int i = 1; i <= kNumberOfSignatures; i++) {
auto builder = exec::FunctionSignatureBuilder()
.knownTypeVariable("K")
.typeVariable("V")
.returnType("map(K,V)");
for (int arg = 0; arg < i; arg++) {
builder.argumentType("K").argumentType("V");
}
signatures.push_back(builder.build());
}
return signatures;
}
```

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start with Velox's functions/sparksql/Map.cpp signature definitions, then trace the reported failure through ExprCompiler.cpp and SubstraitToVeloxPlanValidator.cc. Reproduce the map call with more than three key/value pairs and determine whether the signature limit causes the rejection; done means the reported input validates and executes correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.