alibaba / alibaba/euler

请问是否能提供分布式数据生成的用例?

Open
#50 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

我现在希望从Edge转化到Block,因为我本身也是从分布式的Edge文件转的(而不是JSON文件),我尝试着用Spark Scala写了个demo,但是输出的文件非常小(只有M级别),请问是哪里出了问题么?同样的伪代码在Python本地是work的。

```
val blockRDD = nEdges.map(r => (r.srcId, Set((r.dstId.toLong, r.attr.toFloat)))).reduceByKey(_ ++ _, 1000)
.map { case (nodeId, neighbors: Set[(Long, Float)]) => {
val block = new Block
block.setNode_id(nodeId)
block.setNode_weight(1.0f)
block.setNodeType(0)
val neighbors_ = neighbors.toArray
val neighbor = new java.util.HashMap[Integer, java.util.HashMap[java.lang.Long, java.lang.Float]]()
for (i <- 0 to meta.getEdge_type_num - 1) {
neighbor.put(i, new java.util.HashMap[java.lang.Long, java.lang.Float])
}
val edgeList = new java.util.ArrayList[EdgeItem]()
val parser = new BlockParser(meta)
for (i <- 0 to neighbors_.length - 1) {
val dstId = neighbors_(i)._1
val weight = neighbors_(i)._2
neighbor.get(0).put(dstId, weight)
val edge = new EdgeItem
edge.setSrc_id(nodeId)
edge.setDst_id(dstId)
edge.setWeight(weight)
edgeList.add(edge)
}
block.setNeighbor(neighbor)
block.setEdge(edgeList)
parser.BlockJsonToBytes(block)
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark Scala conversion shown in the issue, especially the reduceByKey pipeline and BlockParser.BlockJsonToBytes call. Compare it with the working Python version and investigate why the generated output is only megabytes. Done means identifying the cause and documenting a reproducible distributed data-generation example.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.