[RFC] [Java] Higher-level "DataFrame"-like API. Lower barrier to entry, increase adoption/audience and productivity.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Based on feedback from mailing list thread here:
- https://lists.apache.org/thread/852btc8tg5gyxglzkrmddts237fpwk8y
The idea being a higher-level API wrapping `VectorSchemaRoot` and `FieldVector` that use Java objects and a row-oriented style for familiarity.
Along with some utilities for manipulating `DataFrame`'s (IE, combine rows from multiple frames with the same schema, convert to a FlightSQL "GetTables" `Schema` object, etc).
I believe this would be tremendously valuable.
Below is an example of a quickly-thrown-together rough idea, just to get the conversation started:
- Full code available at this gist: https://gist.github.com/GavinRay97/c0434574b4516f55da1eebfd4c1519b6
- This code is probably pretty poor and likely doesn't follow Arrow best-practices
## Example Usage
```java
class DataFrameTest {
public static void main(String[] args) {
DataFrame df = DataFrame.create();
df.addColumn("name", MinorType.VARCHAR, false);
df.addColumn("age", MinorType.INT, false);
df.addColumn("weight", MinorType.FLOAT4, false);
df.addRow(Map.of("name", "Alice", "age", 21, "weight", 50.0));
df.addRow(Map.of("name", "Bob", "age", 30, "weight", 60.0));
System.out.println("======= User DataFrame -> VectorSchemaRoot (TSV) =======");
VectorSchemaRoot root = df.toArrowVectorSchemaRoot();
System.out.println(root.contentToTSVString());
assert (root.getRowCount() == 2) : "Expected 2 rows";
assert (root.getSchema().getFields().size() == 3) : "Expected 3 columns";
DataFrame roundtrip = DataFrame.fromArrowVectorSchemaRoot(root);
assert (df.equals(roundtrip)) : "DataFrame equality failed";
System.out.println("======= Roundtrip (DF -> VectorSchemaRoot -> DF) =======");
System.out.println(roundtrip + "\n");
System.out.println("======= FlightSQL GetTables Schema =======");
VectorSchemaRoot flightSchema = new FlightSQLGetTablesSchemaPOJO(
"catalog1", "schema1", "users", "TABLE", df)
.toArrowVectorSchemaRoot();
System.out.println(flightSchema.contentToTSVString());
System.out.println("======= Merge DataFrames =======");
DataFrame df3 = DataFrame.mergeDataFrames(true, df, roundtrip);
System.out.println(df3.toArrowVectorSchemaRoot().contentToTSVString());
assert (df3.rows().size() == df.rows().size() + roundtrip.rows().size()) : "Merge DataFrame failed";
}
}
```
## Output
```java
======= User DataFrame -> VectorSchemaRoot (TSV) =======
name age weight
Alice 21 50.0
Bob 30 60.0
======= Roundtrip (DF -> VectorSchemaRoot -> DF) =======
DataFrame[
columns=[name: Utf8 not null, age: Int(32, true) not null, weight: FloatingPoint(SINGLE) not null],
rows=[{name=Alice, weight=50.0, age=21}, {name=Bob, weight=60.0, age=30}]
]
======= FlightSQL GetTables Schema =======
catalog_name table_schema db_schema_name table_name table_type
catalog1 [B@4bdeaabb schema1 users TABLE
======= Merge DataFrames =======
name age weight
Alice 21 50.0
Bob 30 60.0
Alice 21 50.0
Bob 30 60.0
```
Contributor guide
Research direction
Start with the linked mailing-list discussion and gist, then inspect the existing Java VectorSchemaRoot and FieldVector APIs that the proposal would wrap. Define the supported DataFrame operations and acceptance criteria with maintainers; the issue is not done until the API scope and design are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100