locationtech / locationtech/geotrellis
Investigate RDD.sample behavior change in Spark 3
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.4k
- Forks
- 360
- Avg merge
- 6h 57m
- Merged PRs (30d)
- 12
Description
Describe the bug
One of our tests changed its behavior after upgrading up to Spark 3.x.
The original comment is located here https://github.com/locationtech/geotrellis/pull/3294#discussion_r489823973
This issue is to track down the root cause of the behavior change, explain it and fix if necessary.
To Reproduce
$ ./sbt
$ project spark
$ testOnly geotrellis.spark.summary.StatsTileRDDMethodsSpec -- -z "should be able to sample a fraction of an RDD to compute a histogram"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with spark/src/test/scala/geotrellis/spark/summary/StatsTileRDDMethodsSpec.scala, especially the referenced sample-and-histogram test. Run the provided sbt testOnly command and compare its behavior under Spark 3.x with the earlier behavior described in pull request 3294. Done means the behavior change is explained and fixed if necessary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100