dependency and how to use the jar
- Dominant language
- Scala
- Stars
- 137
- Forks
- 52
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
I would like to know whether the new release is compatible with Spark 3 or not, and how to use the jar file of SparkCube
I tried to compile the code
There's a problem with the following dependency it's unavailable
com.swoop
spark-alchemy_2.11
0.3.28
`Failed to collect dependencies at com.swoop:spark-alchemy_2.11:jar:0.3.28: Failed to read artifact descriptor for com.swoop:spark-alchemy_2.11:jar:0.3.28: Could not transfer artifact com.swoop:spark-alchemy_2.11:pom:0.3.28 from/to swoop-inc (https://dl.bintray.com/swoop-inc/maven/): Access denied to: https://dl.bintray.com/swoop-inc/maven/com/swoop/spark-alchemy_2.11/0.3.28/spark-alchemy_2.11-0.3.28.pom , ReasonPhrase:Forbidden`
If I change the dependency in pom.xml to
spark-alchemy-test_2.12
1.0.1
I have errors (please see below)
```
private implicit def cacheIdToTableIdent(cacheIdentifier: CacheIdentifier): TableIdentifier = {
[WARNING] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:20: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.HyperLogLogInitSimpleAgg
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:44: not found: value HyperLogLogInitSimpleAgg
[ERROR] HyperLogLogInitSimpleAgg(childExpr, relativeSD)
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:22: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.{HyperLogLogCardinality, HyperLogLogMerge}
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:330: not found: value HyperLogLogCardinality
[ERROR] case _: CardinalityAfter => HyperLogLogCardinality(attrs.head)
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:417: not found: value HyperLogLogCardinality
[ERROR] Some(HyperLogLogCardinality(other))
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:509: not found: value HyperLogLogMerge
[ERROR] HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
[ERROR] ^
[WARNING] one warning found
[ERROR] 6 errors found
```
thank you
Contributor guide
Research direction
Start with pom.xml and verify the referenced spark-alchemy dependency and repository availability. Then inspect PreCountDistinctTransformer.scala and GenPlanFromCache.scala, where the missing HyperLogLog symbols are reported. Done means establishing Spark 3 compatibility and documenting a usable dependency or jar path without those compilation errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100