typelevel / typelevel/frameless

Named intermediate Datasets

Open
#188 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement feature
Dominant language
Scala
Stars
895
Forks
135
Avg merge
1d 16h
Merged PRs (30d)
3

Description

case class Foo(bar: Int, baz: String, bal: Boolean)
val ds: TypedDataset[Foo]

This is nicely named and typed! However, after a select, the names are completely lost:

val ds1: TypedDataset[Tuple2[Int, String]] = ds.select(ds('bar), ds('baz))

The best one can do with the current API is to define a new case class for the intermediate representation, and use .as[] to get a ds1 with useful columns names.

Somes idea to workaround this issue:

  • Use a macro to generate a case classes "on the fly", something like this:

    ds.selectNamed(ds('bar), ds('baz))
    // Expends to
    case class FooBarBaz(bar: Int, baz: String)
    ds.selectNamed(ds('bar), ds('baz)).as[FooBarBaz]
    
  • Instead of TupleN, type the resulting Dataset with a shapeless record and update.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the TypedDataset.select API and its TupleN result described in the issue, then compare the proposed macro-generated case class and shapeless-record approaches. Done means an intermediate selection such as ds.select(ds('bar), ds('baz)) preserves usable column names without requiring a manually defined case class or .as conversion; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.