[Umbrella][Feature] Support for complex data types
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Motivation
Support for complex data types: Array ([#168](https://github.com/alibaba/fluss/issues/168)), Map ([#169](https://github.com/alibaba/fluss/issues/169)), Struct ([#170](https://github.com/alibaba/fluss/issues/170)), Variant/JSON.
**Task list**
- [x] Support for Row Type Foundation @XuQianJin-Stars
- [x] Support for Array data type @XuQianJin-Stars
- [x] Support for Map data type @XuQianJin-Stars
- [x] Support for Struct data type @XuQianJin-Stars
- [ ] Support for Variant/JSON data type
### Solution
1. The interface layer will implement: `InternalArray`, `InternalMap`, and `InternalRow` (to be refined).
2. Add generic implementations: `GenericArray`, `GenericMap`, and `GenericRow` (to be refined).
3. Implement `ArrayType`, `MapType`, `StructType`, `Variant/JSON`, etc., based on Arrow (`ColumnarRow`), `IndexedRow`, and `CompactedRow` respectively.
4. Integrate and transform with Flink/Spark data types.
5. Integrate and transform with data lakes (such as Paimon).
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the remaining Variant/JSON item in the task list and review the proposed InternalArray, InternalMap, and InternalRow interfaces alongside the generic implementations and Arrow-based types described in the solution. Trace the planned Flink/Spark and data-lake integrations, then define completion as Variant/JSON support integrated across the listed layers and reflected by a completed task-list item.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100