typelevel / typelevel/frameless
Easier withColumn method
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
Great work on this lib! It's a great way to write Spark code!
As discussed here and in the docs, withColumn requires a full schema when a column is added.
Here's the example in the docs:
case class CityBedsOther(city: String, bedrooms: Int, other: List[String])
cityBeds.
withColumn[CityBedsOther](lit(List("a","b","c"))).
show(1).run()
Couldn't we just assume that the schema stays the same for the existing columns and only supply the schema for the column that's being added?
cityBeds.
withColumn[List[String]](lit(List("a","b","c"))).
show(1).run()
I think this'd be a lot more use friendly. I'm often dealing with schemas that have tons of columns and add lots of columns with withColumn. Let me know your thoughts!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the discussion in issue 321 and the FeatureOverview documentation linked in this issue, then locate the existing withColumn API in the Frameless source. Compare the current full-schema example with the proposed column-only call; done means the simpler API is supported and its behavior is covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100