linkedin / linkedin/transport

Type safe API for Transport

Open
#8 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
306
Forks
75
PR merge metrics
No merged PRs in 30d

Description

This ticket tries to make a case for a more type safe API for `transport` . Our API today IMO lacks in two ways :-

1. The containers API - Struct, Map and Array is not parameterized. This means that for consuming elements from these containers requires a typecast. E.g taking a `Int` out of a `map` would mean taking `StdData` out and then typecasting it to a `StdInt`.
1. The StdUDF API is parameterized, but the parameters extend StdData. This is limiting because :-
1. Supporting #7 would require typecasting
2. It would make it impossible to support #6

I think we can do better, and though Presto may have some unknowns, I was able to achieve some success with a Spark prototype. Below I try to give an idea of my approach

**Containers API**
As described in #7, I chose `java.util.{List, Map}` for List and Map types. For Struct I defined a `Record` type, similar in line to Avro's `GenericRecord`. The key point here is the all these container APIs are parameterized as shown below
```scala
trait Schema {
def schema: DataType
}

trait IndexedRecord extends Schema {
def put[V](i: Int, v: V): Unit

def get[V](i: Int): V
}

trait GenericRecord extends IndexedRecord {
def put[V](key: String, v: V): Unit

def get[V](key: String): V
}

abstract class GenericList[A] extends util.AbstractList[A] with Schema

abstract class GenericMap[K, V] extends util.AbstractMap[K, V] with Schema
```
So to get a field out of a record, we'd do
```java
final GenericRecord r = ...
final List f = r.get("A");
```
This does not involve any typecasting and is type safe. Similar examples can be given for other container types.

**UDF API**
Similarly, for UDF API, we can have generic parameters which need not extend `StdData` . Below I provide the API I used in my prototype.
```scala
trait Fn0[F] extends UDF0[F] with Fn

trait Fn1[T, F] extends UDF1[T, F] with Fn

trait Fn2[T1, T2, F] extends UDF2[T1, T2, F] with Fn

trait Fn3[T1, T2, T3, F] extends UDF3[T1, T2, T3, F] with Fn

trait Fn4[T1, T2, T3, T4, F] extends UDF4[T1, T2, T3, T4, F] with Fn

trait Fn5[T1, T2, T3, T4, T5, F] extends UDF5[T1, T2, T3, T4, T5, F] with Fn

trait Fn6[T1, T2, T3, T4, T5, T6, F] extends UDF6[T1, T2, T3, T4, T5, T6, F] with Fn
```
This would help us implement #7 cleanly and in a type safe manner, and would make #6 possible

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the transport containers API and StdUDF API described in the issue, then compare the proposed generic interfaces with the Spark prototype and the supported Spark, Hive, and Presto engines. Done means agreeing on a concrete type-safe API design and validating that it addresses the container and UDF limitations without leaving compatibility requirements unspecified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.