mapstruct / mapstruct/mapstruct-examples
Convert Spark Dataframe to Dataset with MapStruct
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 504
- PR merge metrics
- No merged PRs in 30d
Description
When using Apache Spark with Java there is a pretty common use case of converting Spark's Dataframes to POJO-based Datasets. The thing is that many times your Dataframe is imported from a database in which the column namings and types are different from your POJO.
Example for this can be found on the following [Stackoverflow question](https://stackoverflow.com/questions/69802146/convert-spark-df-to-a-ds-with-different-fields-names) (which has been solved programmatically with a map and rename functions). I was wondering if that example could be solved with MapStruct (and maybe would be added to this examples repo).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the mapstruct-examples repository and the linked Stack Overflow question to understand the requested Spark DataFrame-to-Dataset conversion. Identify where a Spark and Java example would fit, then determine whether MapStruct can handle differing database column names and POJO field types. Done means a documented, working example covering that conversion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100