typelevel / typelevel/frameless
Named intermediate Datasets
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
case class Foo(bar: Int, baz: String, bal: Boolean)
val ds: TypedDataset[Foo]
This is nicely named and typed! However, after a select, the names are completely lost:
val ds1: TypedDataset[Tuple2[Int, String]] = ds.select(ds('bar), ds('baz))
The best one can do with the current API is to define a new case class for the intermediate representation, and use .as[] to get a ds1 with useful columns names.
Somes idea to workaround this issue:
-
Use a macro to generate a case classes "on the fly", something like this:
ds.selectNamed(ds('bar), ds('baz)) // Expends to case class FooBarBaz(bar: Int, baz: String) ds.selectNamed(ds('bar), ds('baz)).as[FooBarBaz] -
Instead of
TupleN, type the resulting Dataset with a shapeless record and update.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the TypedDataset.select API and its TupleN result described in the issue, then compare the proposed macro-generated case class and shapeless-record approaches. Done means an intermediate selection such as ds.select(ds('bar), ds('baz)) preserves usable column names without requiring a manually defined case class or .as conversion; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100