[FEA] Test map with map as key

Open
#9,155 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
scala
Domain
data-engineering

Research direction

No files or existing tests are named; start by locating the current checks for map-key support and the implementations of the listed operators. Add coverage for operators that support maps as keys, including repartition and ORC/Parquet reads and writes, and document or test the unsupported literal and map-construction cases.

Written by the indexing model from the issue text.

Description

test

Is your feature request related to a problem? Please describe.
Spark doesn't really support maps with maps as the key. You can force it to happen with something like.

val df  = Seq((Map(Map(1 -> 1) -> 1), Map(Map(2 -> 1) -> 2)), (Map(Map(2 -> 2) -> 0), Map(Map(2 -> 2) -> 0))).toDF("a", "b")

But once you have that spark is not really all that happy to let you do anything with it. That is because a map key has to be orderable to do much with it. But we say that we support it in a lot of our checks, so it would be good to go though as many operators as we can and add tests for the ones that we can support.

Operators that we should test...

  • map_entries
  • isNull
  • isNotNull
  • coalesce
  • if
  • first
  • last
  • getStructField
  • getArrayItem
  • map_values
  • map_keys
  • map_filter
  • explode
  • explode_outer
  • size
  • repartition (both hash and round robin)
  • transform_keys (so long as the key returned is not also a map)
  • transform_values
  • file writes (ORC, Parquet)
  • file reads (ORC, Parquet)

Operators that we say we support, but spark does not...

  • literal (not 100% sure on that, Spark does not really support maps as a lit at all when trying to go through the front door)
  • map
  • map_from_arrays
  • map_from_entries

To be clear this is not a high priority because Spark does not really support this all that well anyways, and I think we are doing the right thing is all the cases I manually tested.

Dominant language
Scala
Stars
1k
Forks
305
Avg merge
3d 12h
Merged PRs (30d)
148

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/cudf-spark

All issues in NVIDIA/cudf-spark

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.