alibaba / alibaba/SparkCube

dependency and how to use the jar

Open
#17 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
137
Forks
52
PR merge metrics
No merged PRs in 30d

Description

Hi all,
I would like to know whether the new release is compatible with Spark 3 or not, and how to use the jar file of SparkCube
I tried to compile the code
There's a problem with the following dependency it's unavailable

com.swoop
spark-alchemy_2.11
0.3.28

`Failed to collect dependencies at com.swoop:spark-alchemy_2.11:jar:0.3.28: Failed to read artifact descriptor for com.swoop:spark-alchemy_2.11:jar:0.3.28: Could not transfer artifact com.swoop:spark-alchemy_2.11:pom:0.3.28 from/to swoop-inc (https://dl.bintray.com/swoop-inc/maven/): Access denied to: https://dl.bintray.com/swoop-inc/maven/com/swoop/spark-alchemy_2.11/0.3.28/spark-alchemy_2.11-0.3.28.pom , ReasonPhrase:Forbidden`

If I change the dependency in pom.xml to
spark-alchemy-test_2.12
1.0.1
I have errors (please see below)

```
private implicit def cacheIdToTableIdent(cacheIdentifier: CacheIdentifier): TableIdentifier = {
[WARNING] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:20: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.HyperLogLogInitSimpleAgg
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:44: not found: value HyperLogLogInitSimpleAgg
[ERROR] HyperLogLogInitSimpleAgg(childExpr, relativeSD)
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:22: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.{HyperLogLogCardinality, HyperLogLogMerge}
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:330: not found: value HyperLogLogCardinality
[ERROR] case _: CardinalityAfter => HyperLogLogCardinality(attrs.head)
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:417: not found: value HyperLogLogCardinality
[ERROR] Some(HyperLogLogCardinality(other))
[ERROR] ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:509: not found: value HyperLogLogMerge
[ERROR] HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
[ERROR] ^
[WARNING] one warning found
[ERROR] 6 errors found
```
thank you

Contributor guide

Open the contributing guide

Research direction

Start with pom.xml and verify the referenced spark-alchemy dependency and repository availability. Then inspect PreCountDistinctTransformer.scala and GenPlanFromCache.scala, where the missing HyperLogLog symbols are reported. Done means establishing Spark 3 compatibility and documenting a usable dependency or jar path without those compilation errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.