typelevel / typelevel/frameless
Cannot use colMany to access anything wrapped in Option
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
My understanding is that Option should be used to represent columns that one might mark nullable in vanilla Spark. I tried something along the lines of the following:
case class B(i: Int)
case class A(ob: Option[B], b: B)
implicit val sqlc = spark.sqlContext
val as = TypedDataset.create(Seq(
A(Some(B(1)), B(2)),
A(None, B(3))))
as.colMany('b, 'i) // works
as.colMany('ob, 'i) // runs into the option and can't find i as a field in it?
The last line resulted in
<console>:27: error: No columns shapeless.::[shapeless.tag.@@[Symbol,String("ob")],shapeless.::[shapeless.tag.@@[Symbol,String("i")],shapeless.HNil]] of type Out in A
What I think is reasonable is to return something of type TypedColumn[A, Option[Int]]. For comparison, in regular Spark:
scala> as.dataset.select("ob.i").show
+----+
| _1|
+----+
| 1|
|null|
+----+
which I would expect as.select(as.colMany('ob, 'i).show().run to be roughly equivalent to up to some decisions on whether to display null or None.
Perhaps a reasonable way to approach this problem is to integrate the column selection mechanism with some kind of optics.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the colMany calls and the nested Option example in the issue, then trace the column selection mechanism used for nested fields. Compare its behavior with Spark's "ob.i" selection and determine how the requested TypedColumn[A, Option[Int]] result should be represented. Done means nested fields inside Option can be selected with the expected nullable result, with coverage for both Some and None cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100