apache / apache/geaflow

Streaming composition Traversal, data cannot be inserted

Open
#575 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
808
Forks
188
Avg merge
3d 22h
Merged PRs (30d)
2

Description

**Describe the bug**
A clear and concise description of what the bug is.
Streaming composition Traversal, data cannot be inserted,As a result, the cyclic trading results cannot be reproduced.
**Expected behavior**
A clear and concise description of what you expected to happen.
2025-08-01 14:17:51,517 [shuffle-reader-7[0/2]] INFO PipelineInputFetcher:211 - fetcher task finish window id 884 shuffle-reader-7[0/2]
2025-08-01 14:17:51,879 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#1[1], insert fail, try insert
2025-08-01 14:17:51,883 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#1[8], insert fail, try insert
2025-08-01 14:17:51,886 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#1[9], insert fail, try insert
2025-08-01 14:17:51,890 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#1[13], insert fail, try insert
2025-08-01 14:17:51,894 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#1[14], insert fail, try insert
2025-08-01 14:17:51,897 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#2-1[15], insert fail, try insert
2025-08-01 14:17:51,901 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#2-2[15], insert fail, try insert
2025-08-01 14:17:51,905 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#2-3[15], insert fail, try insert
2025-08-01 14:17:51,909 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#2-4[15], insert fail, try insert
2025-08-01 14:17:51,913 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#2-5[15], insert fail, try insert
2025-08-01 14:17:51,917 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_CYCLE_PipelineTask#1-884_cycle#3[18], insert fail, try insert
2025-08-01 14:17:51,924 [stats-flusher-0] INFO JdbcKVStore:74 - key: geaflow1753927737412687237_metrics_PIPELINE_PipelineTask#1-885, insert fail, try insert
2025-08-01 14:17:51,928 [stats-flusher-0] INFO BaseJdbcStore:148 - flush
2025-08-01 14:17:52,931 [stats-flusher-0] INFO BaseJdbcStore:148 - flush
2025-08-01 14:18:21,431 [geaflow-worker-2] INFO GeaFlowTableSourceFunction:202 - fetch data size: 0, isFinish: false, table: table_new_trade, partition: tbl-trade-0, batchId: 885,nextOffset: 2025-08-01 14:18:21, cost 30008
2025-08-01 14:18:21,431 [geaflow-worker-2] INFO KafkaConsumer:1123 - [Consumer clientId=consumer-default-group-id-3, groupId=default-group-id] Subscribed to partition(s): tbl-trade-2
2025-08-01 14:18:21,431 [geaflow-worker-2] INFO KafkaConsumer:1564 - [Consumer clientId=consumer-default-group-id-3, groupId=default-group-id] Seeking to offset 0 for partition tbl-trade-2
2025-08-01 14:18:22,940 [stats-flusher-0] INFO BaseJdbcStore:148 - flush
2025-08-01 14:18:51,439 [geaflow-worker-2] INFO GeaFlowTableSourceFunction:202 - fetch data size: 0, isFinish: false, table: table_new_trade, partition: tbl-trade-2, batchId: 885,nextOffset: 2025-08-01 14:18:51, cost 30008
2025-08-01 14:18:51,439 [geaflow-worker-2] INFO KafkaConsumer:1123 - [Consumer clientId=consumer-default-group-id-3, groupId=default-group-id] Subscribed to partition(s): tbl-trade-1
2025-08-01 14:18:51,439 [geaflow-worker-2] INFO KafkaConsumer:1564 - [Consumer clientId=consumer-default-group-id-3, groupId=default-group-id] Seeking to offset 0 for partition tbl-trade-1
2025-08-01 14:18:52,946 [stats-flusher-0] INFO BaseJdbcStore:148 - flush
**Additional context**
Add any other context about the problem here.
https://tugraph-analytics.github.io/opinion/2023/06/16/%E8%B0%81%E5%9C%A8%E4%BB%A5%E5%A4%AA%E5%9D%8A%E5%8C%BA%E5%9D%97%E9%93%BE%E4%B8%8A%E5%BE%AA%E7%8E%AF%E4%BA%A4%E6%98%93-TuGraph+Kafka%E7%9A%840%E5%85%83%E6%B5%81%E5%9B%BE%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88.html

Contributor guide

Open the contributing guide

Research direction

Start by tracing the logged failure through PipelineInputFetcher, JdbcKVStore, and BaseJdbcStore, then inspect GeaFlowTableSourceFunction and KafkaConsumer around the repeated partition reads. Reproduce the streaming composition Traversal with the cyclic trading workload and verify that data is inserted and the cyclic trading results can be reproduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.