[Improvement] Support UserDefinedType for Spark 3.2 and above
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
As user-defined type (UDT) API is public since Spark 3.2.0, kyuubi should support udt schema converter.
reference:
[SPARK-7768](https://issues.apache.org/jira/browse/SPARK-7768)
### How should we improve?
Add udt to SchemaHelper.
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Locate SchemaHelper and read how it currently converts Spark schemas, then review the Spark 3.2 UserDefinedType API and the SPARK-7768 reference. Done means SchemaHelper supports UDT schemas on Spark 3.2 and above, with the relevant behavior covered by the project's existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100