apache / apache/hugegraph-toolchain

[Bug] Spark Loader insertSuccessCnt statistics incorrect

Open
#502 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
121
Forks
125
Avg merge
10d 10h
Merged PRs (30d)
3

Description

### Bug Type (问题类型)

others (please comment below)

Spark Loader success but ` insertSuccessCnt` is not right, always be 0.

### Before submit

- [X] I had searched in the [issues](https://github.com/apache/hugegraph-toolchain/issues) and found no similar issues.

### Environment (环境信息)

- Server Version: v1.0.0
- Toolchain Version: master
- Data Size: xx vertices, xx edges

### Expected & Actual behavior (期望与实际表现)

```
23/08/04 00:11:49 INFO HugeGraphSparkLoader:
Finished load example/spark/vertex_software.json data
23/08/04 00:11:49 INFO DAGScheduler: Job 5 finished: foreachPartition at HugeGraphSparkLoader.java:154, took 1.023988 s
23/08/04 00:11:49 INFO DAGScheduler: ResultStage 3 (foreachPartition at HugeGraphSparkLoader.java:154) finished in 1.021 s
23/08/04 00:11:49 INFO HugeGraphSparkLoader:
Finished load example/spark/edge_knows.json data
23/08/04 00:11:49 INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job
23/08/04 00:11:49 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished
23/08/04 00:11:49 INFO DAGScheduler: Job 4 finished: foreachPartition at HugeGraphSparkLoader.java:154, took 1.025300 s
23/08/04 00:11:49 INFO HugeGraphSparkLoader:
Finished load example/spark/vertex_person.json data
23/08/04 00:11:49 INFO HugeGraphSparkLoader:
------------The data load task is complete-------------------

insertSuccessCnt: 0
---------------------------------------------

23/08/04 00:11:49 INFO SparkUI: Stopped Spark web UI at http://192.168.34.164:4040
23/08/04 00:11:49 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/08/04 00:11:49 INFO MemoryStore: MemoryStore cleared
23/08/04 00:11:49 INFO BlockManager: BlockManager stopped
23/08/04 00:11:49 INFO BlockManagerMaster: BlockManagerMaster stopped
23/08/04 00:11:49 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/08/04 00:11:49 INFO SparkContext: Successfully stopped SparkContext
23/08/04 00:11:49 INFO SparkContext: SparkContext already stopped.
23/08/04 00:11:49 INFO SparkContext: SparkContext already stopped.
23/08/04 00:12:49 INFO ShutdownHookManager: Shutdown hook called
23/08/04 00:12:49 INFO ShutdownHookManager: Deleting directory /tmp/spark-41fba547-151e-4a5a-8982-fa9df80978db
23/08/04 00:12:49 INFO ShutdownHookManager: Deleting directory /tmp/spark-8c9b8b45-a9cf-4876-a719-b7446f4e46cd
```

### Vertex/Edge example (问题点 / 边数据举例)

_No response_

### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

```javascript
// Define schema
schema.propertyKey("name").asText().ifNotExist().create();
schema.propertyKey("age").asInt().ifNotExist().create();
schema.propertyKey("city").asText().ifNotExist().create();
schema.propertyKey("weight").asDouble().ifNotExist().create();
schema.propertyKey("lang").asText().ifNotExist().create();
schema.propertyKey("date").asText().ifNotExist().create();
schema.propertyKey("price").asDouble().ifNotExist().create();

schema.vertexLabel("person")
.properties("name", "age", "city")
.useCustomizeStringId()
.nullableKeys("age", "city")
.ifNotExist()
.create();

schema.vertexLabel("software")
.properties("name", "lang", "price")
.useCustomizeStringId()
.ifNotExist()
.create();

schema.edgeLabel("knows")
.sourceLabel("person")
.targetLabel("person")
.properties("date", "weight")
.ifNotExist()
.create();

schema.edgeLabel("created")
.sourceLabel("person")
.targetLabel("software")
.properties("date", "weight")
.ifNotExist()
.create();
```

```
{
"vertices": [
{
"label": "person",
"input": {
"type": "file",
"path": "example/spark/vertex_person.json",
"format": "JSON",
"header": ["name", "age", "city"],
"charset": "UTF-8",
"skipped_line": {
"regex": "(^#|^//).*"
}
},
"id": "name",
"null_values": ["NULL", "null", ""]
},
{
"label": "software",
"input": {
"type": "file",
"path": "example/spark/vertex_software.json",
"format": "JSON",
"header": ["id","name", "lang", "price","ISBN"],
"charset": "GBK"
},
"id": "name",
"ignored": ["ISBN"]
}
],
"edges": [
{
"label": "knows",
"source": ["source_name"],
"target": ["target_name"],
"input": {
"type": "file",
"path": "example/spark/edge_knows.json",
"format": "JSON",
"date_format": "yyyyMMdd",
"header": ["source_name","target_name", "date", "weight"]
},
"field_mapping": {
"source_name": "name",
"target_name": "name"
}
}
]
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HugeGraphSparkLoader.java around the foreachPartition call at line 154 and trace how successful inserts are counted and reported. Reproduce the load using the referenced vertex and edge example files, then verify that insertSuccessCnt reflects the completed inserts instead of remaining zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.