typelevel / typelevel/frameless

Cannot use colMany to access anything wrapped in Option

Open
#168 8 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Scala
Stars
895
Forks
135
Avg merge
1d 16h
Merged PRs (30d)
3

Description

My understanding is that Option should be used to represent columns that one might mark nullable in vanilla Spark. I tried something along the lines of the following:

case class B(i: Int)
case class A(ob: Option[B], b: B)

implicit val sqlc = spark.sqlContext
val as = TypedDataset.create(Seq(
  A(Some(B(1)), B(2)),
  A(None, B(3))))

as.colMany('b, 'i) // works
as.colMany('ob, 'i) // runs into the option and can't find i as a field in it?

The last line resulted in

<console>:27: error: No columns shapeless.::[shapeless.tag.@@[Symbol,String("ob")],shapeless.::[shapeless.tag.@@[Symbol,String("i")],shapeless.HNil]] of type Out in A

What I think is reasonable is to return something of type TypedColumn[A, Option[Int]]. For comparison, in regular Spark:

scala> as.dataset.select("ob.i").show
+----+
|   _1|
+----+
|   1|
|null|
+----+

which I would expect as.select(as.colMany('ob, 'i).show().run to be roughly equivalent to up to some decisions on whether to display null or None.
Perhaps a reasonable way to approach this problem is to integrate the column selection mechanism with some kind of optics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the colMany calls and the nested Option example in the issue, then trace the column selection mechanism used for nested fields. Compare its behavior with Spark's "ob.i" selection and determine how the requested TypedColumn[A, Option[Int]] result should be represented. Done means nested fields inside Option can be selected with the expected nullable result, with coverage for both Some and None cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.