4paradigm / 4paradigm/OpenMLDB

external udf in offline

Open
#3,377 1 comment 0 reactions 0 assignees View on GitHub
batch-engine bug sql
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

In batch, udf register is weird, need fix.
- udf load
https://github.com/4paradigm/OpenMLDB/blob/29b74f721e12b09142413b58a71112e5925aaaf2/java/openmldb-batch/src/main/scala/com/_4paradigm/openmldb/batch/SparkPlanner.scala#L352-L371
If file in proto is abs path, e.g. /tmp/libudf.so, it cuts to libudf.so and use `/libudf.so` to register
in local mode(yarn mode will use libudf.so, and now we'll use `/libudf.so` to load it, it's ok)
- still in local mode, openmldbCatalogService will register udf first, then engine in sparkplanner will register again, so in local mode the abs udf register succeed in the first time, failed in second time, it's confused. Test should be caution.

**No offline udf test** (local & yarn), `create function` only effects taskmanager, we should run the job to check udf load in batch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.