[VL] Fallback for map function
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
I see this fallback when using map function:
(25) Project: Native validation failed:
Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1381 function:validate, thrown from file:ExprCompiler.cpp line:477 function:compileRewrittenExpression, reason:Scalar function map not registered with arguments: (VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE, VARCHAR, DOUBLE). Found function registered with the following signatures:
((K,V) -> map(K,V))
((K,V,K,V) -> map(K,V))
((K,V,K,V,K,V) -> map(K,V))
Is it because of this signature definition in velox here https://github.com/facebookincubator/velox/blob/00d2ca375016318d5a4cc061775e462c4f4f555e/velox/functions/sparksql/Map.cpp#L117C2-L136C15? Can we just increase it to support more than 6 inputs by chanigng the value to more than 3? @PHILO-HE
```
static std::vector> signatures() {
// For the purpose of testing we introduce up to 6 inputs
// array(K), array(V) -> map(K,V)
std::vector> signatures;
constexpr int kNumberOfSignatures = 3;
signatures.reserve(kNumberOfSignatures);
for (int i = 1; i <= kNumberOfSignatures; i++) {
auto builder = exec::FunctionSignatureBuilder()
.knownTypeVariable("K")
.typeVariable("V")
.returnType("map(K,V)");
for (int arg = 0; arg < i; arg++) {
builder.argumentType("K").argumentType("V");
}
signatures.push_back(builder.build());
}
return signatures;
}
```
### Gluten version
main branch
Contributor guide
Research direction
Start with Velox's functions/sparksql/Map.cpp signature definitions, then trace the reported failure through ExprCompiler.cpp and SubstraitToVeloxPlanValidator.cc. Reproduce the map call with more than three key/value pairs and determine whether the signature limit causes the rejection; done means the reported input validates and executes correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100