apache / apache/gluten

[VL] Allow a native Velox UDF to be used without a matching Java Hive UDF class

Open
#13,014 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

**Proposal: Infer the Java signature from the Velox C++ UDF**

**Why?** The velox C++ UDF implementation already contains the necessary information needed to derive the signature, defining the java class introduces an extra step for the user that can be avoided. From a user experience point of view, user can then write a Velox C++ udf once and have it available to different backends out of the box (i.e they get spark out of the box for free).

The following section shares the current steps needed to define a UDF. After this proposal:
- Step1 would no longer be needed
- Step4 would no longer be needed
- Step5 would no longer need the `CREATE TEMPORARY FUNCTION my_new_udf AS 'com.example.udf.MyNewUdf';`
- Step6 would no longer need the `--jars "$BUNDLE",out/mynewudf.jar ` and ` --driver-class-path "$BUNDLE:out/mynewudf.jar" `

## Current State

- A UDF that only ever runs on Gluten needs a Java class written, compiled, jarred and shipped purely as a name.
- The Java body and the C++ body must be kept in sync by hand; nothing enforces it.
- Any change to the signature has to be made twice, in two languages.

### [step1] Write the Java UDF

> Not needed after this proposal.

```java
package com.example.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class MyNewUdf extends UDF {

public Text evaluate(Text a, Text b) {
throw new UnsupportedOperationException(
"com.example.udf.MyNewUdf is implemented natively in Velox; "
+ "reaching this method means the query fell back to the JVM.");
}
}

```

### [step2] Write the C++ Velox UDF

Put `MyNewUdf.cc` in `cpp/velox/udf/examples/` so it inherits the flags the shipped examples use.

```cpp
#include
#include
#include "udf/Udf.h"
#include "udf/examples/UdfCommon.h" // gluten::UdfRegisterer

using namespace facebook::velox;

namespace {
static const char* kVarChar = "varchar";

template
struct MyNewUdfFunction {
VELOX_DEFINE_FUNCTION_TYPES(T);
FOLLY_ALWAYS_INLINE void call(
out_type& result,
const arg_type& a,
const arg_type& b) {
result.append(a.data());
result.append(" ");
result.append(b.data());
}
};

class MyNewUdfRegisterer final : public gluten::UdfRegisterer {
public:
int getNumUdf() override { return 1; }

void populateUdfEntries(int& i, gluten::UdfEntry* e) override {
e[i++] = {name_.c_str(), kVarChar, 2, arg_, false, true};
}

void registerSignatures() override {
registerFunction({name_});
}

private:
const std::string name_ = "com.example.udf.MyNewUdf";
const char* arg_[2] = {kVarChar, kVarChar};
};
} // namespace
...
// globalRegisters(), setupRegisterers(), and the DEFINE_GET_NUM_UDF /
// DEFINE_GET_UDF_ENTRIES / DEFINE_REGISTER_UDF macros: copy verbatim from MyUDF.cc.
```

`name_` must equal the Java class name — that is the binding this proposal removes.

References: [MyUDF.cc](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/MyUDF.cc), [UdfCommon.h](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/UdfCommon.h), [Udf.h](https://github.com/apache/gluten/blob/main/cpp/velox/udf/Udf.h), [VeloxUDF.md](https://github.com/apache/gluten/blob/main/docs/developers/VeloxUDF.md)

### [step3] Build the `.so`

Append to [cpp/velox/udf/examples/CMakeLists.txt](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/CMakeLists.txt):

```cmake
add_library(mynewudf SHARED "MyNewUdf.cc")
target_link_libraries(mynewudf velox)
```

```bash
docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c '
source /opt/rh/gcc-toolset-12/enable
export VELOX_BUILD_SHARED=ON
cd /work/gluten && ./dev/builddeps-veloxbe.sh --build_examples=ON
'
```

`--build_examples=ON` is what pulls `udf/examples` into the build. Incremental on an already-built tree: 24 seconds. Produces `cpp/build/velox/udf/examples/libmynewudf.so`.

Use the `centos-9` image; `centos-8` ships an fbthrift predating `readLEFromBuffer` and Velox's Parquet reader will not compile against it.

### [step4] Compile the Java UDF into a jar

> Not needed after this proposal.

```bash
javac -cp "$SPARK_HOME/jars/*" -d out/classes MyNewUdf.java
jar cf out/mynewudf.jar -C out/classes .
```

### [step5] Write the query

> After this proposal the `CREATE TEMPORARY FUNCTION` line would no longer be needed.

```sql
CREATE TEMPORARY FUNCTION my_new_udf AS 'com.example.udf.MyNewUdf';

CREATE TABLE IF NOT EXISTS greetings AS
SELECT * FROM VALUES ('hello'), ('bonjour'), ('hola') AS t(word);

SELECT word, my_new_udf(word, 'world') AS greeting FROM greetings ORDER BY word;

EXPLAIN SELECT my_new_udf(word, 'world') FROM greetings;
```

### [step6] Run the query

> After this proposal the bundle jar would no longer need to go on the classpath.

```bash
$SPARK_HOME/bin/spark-sql \
--master 'local[2]' \
--jars "$BUNDLE",out/mynewudf.jar \
--driver-class-path "$BUNDLE:out/mynewudf.jar" \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=4g \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///abs/path/libmynewudf.so \
--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///abs/path/libmynewudf.so \
--conf spark.sql.catalogImplementation=hive \
-f demo.sql
```

#### Results

Verified end to end on 2026-09-13 against `main` at commit `84a2b7a`, Spark 3.5.9.

```
bonjour bonjour world
hello hello world
hola hola world
Time taken: 1.789 seconds, Fetched 3 row(s)
```

```
== Physical Plan ==
VeloxColumnarToRow
+- ^(1) ProjectExecTransformer [HiveSimpleUDF#com.example.udf.MyNewUdf(word#12,world) AS my_new_udf(word, world)#13]
+- ^(1) InputIteratorTransformer[word#12]
+- RowToVeloxColumnar
+- Scan hive spark_catalog.default.greetings [word#12], HiveTableRelation [`spark_catalog`.`default`.`greetings`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [word#12], Partition Cols: []]
```

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start by reading cpp/velox/udf/examples/MyUDF.cc, udf/examples/UdfCommon.h, udf/Udf.h, and docs/developers/VeloxUDF.md to understand registration and signature handling. Review cpp/velox/udf/examples/CMakeLists.txt as the build entry point. Done means a native Velox UDF can be registered and used without a matching Java UDF class or Java bundle, with its signature inferred from the C++ implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, docker, java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.