linkedin / linkedin/transport

Transport support for existing udfs written in presto for execution in hive/spark etc

Open
#9 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
306
Forks
75
PR merge metrics
No merged PRs in 30d

Description

Presto seems to support lot of rich functions that we find very useful, examples are:

- string functions - https://prestodb.github.io/docs/current/functions/string.html
- geospatial function - https://prestodb.github.io/docs/current/functions/geospatial.html
- url functions - https://prestodb.github.io/docs/current/functions/url.html

These are just few of all the functions supported by presto.

If transport also had apis to translate presto functions to transport udf, it could unlock the rich set of presto functions to other execution engines (to spark or flink for example). A distro of presto out-of-the-box functions made available using transport for hive/flink can be of immense value to users.

Extending this idea, if the same is done to support hive udfs, that would help adoption as teams who already has lot of hive udfs can take advantage of the framework right away without having to re-write those udfs using transport udf.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review Transport's UDF APIs and its integrations with execution engines before scoping support for existing Presto and Hive UDFs. Done would mean a defined way to translate or expose those UDFs for engines such as Spark and Flink, with the supported function coverage and compatibility boundaries documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.