Type safe API for Transport
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 306
- Forks
- 75
- PR merge metrics
- No merged PRs in 30d
Description
This ticket tries to make a case for a more type safe API for `transport` . Our API today IMO lacks in two ways :-
1. The containers API - Struct, Map and Array is not parameterized. This means that for consuming elements from these containers requires a typecast. E.g taking a `Int` out of a `map` would mean taking `StdData` out and then typecasting it to a `StdInt`.
1. The StdUDF API is parameterized, but the parameters extend StdData. This is limiting because :-
1. Supporting #7 would require typecasting
2. It would make it impossible to support #6
I think we can do better, and though Presto may have some unknowns, I was able to achieve some success with a Spark prototype. Below I try to give an idea of my approach
**Containers API**
As described in #7, I chose `java.util.{List, Map}` for List and Map types. For Struct I defined a `Record` type, similar in line to Avro's `GenericRecord`. The key point here is the all these container APIs are parameterized as shown below
```scala
trait Schema {
def schema: DataType
}
trait IndexedRecord extends Schema {
def put[V](i: Int, v: V): Unit
def get[V](i: Int): V
}
trait GenericRecord extends IndexedRecord {
def put[V](key: String, v: V): Unit
def get[V](key: String): V
}
abstract class GenericList[A] extends util.AbstractList[A] with Schema
abstract class GenericMap[K, V] extends util.AbstractMap[K, V] with Schema
```
So to get a field out of a record, we'd do
```java
final GenericRecord r = ...
final List f = r.get("A");
```
This does not involve any typecasting and is type safe. Similar examples can be given for other container types.
**UDF API**
Similarly, for UDF API, we can have generic parameters which need not extend `StdData` . Below I provide the API I used in my prototype.
```scala
trait Fn0[F] extends UDF0[F] with Fn
trait Fn1[T, F] extends UDF1[T, F] with Fn
trait Fn2[T1, T2, F] extends UDF2[T1, T2, F] with Fn
trait Fn3[T1, T2, T3, F] extends UDF3[T1, T2, T3, F] with Fn
trait Fn4[T1, T2, T3, T4, F] extends UDF4[T1, T2, T3, T4, F] with Fn
trait Fn5[T1, T2, T3, T4, T5, F] extends UDF5[T1, T2, T3, T4, T5, F] with Fn
trait Fn6[T1, T2, T3, T4, T5, T6, F] extends UDF6[T1, T2, T3, T4, T5, T6, F] with Fn
```
This would help us implement #7 cleanly and in a type safe manner, and would make #6 possible
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the transport containers API and StdUDF API described in the issue, then compare the proposed generic interfaces with the Spark prototype and the supported Spark, Hive, and Presto engines. Done means agreeing on a concrete type-safe API design and validating that it addresses the container and UDF limitations without leaving compatibility requirements unspecified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100