[issue] In effectively_once mode, when a single pod fails, the entire function will fail due to failure to create the producer.
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- kubernetes
- Domain
- distributed-systems, stream-processing
Research direction
Start by reproducing effectively_once mode with five Kubernetes pods and fail one pod, then inspect producer-creation and failover logs. Also reproduce the PersistenceError case; done means subscription failover no longer causes all function pods to restart and the function does not remain stuck.
Written by the indexing model from the issue text.
Description
- When I enable effectively_once and deploy the function in k8s, for example, after deploying 5 pods, when one of my pods crashes, his subscription will be transferred to other pods due to failover mode. At this time, other pods will fail to create the producer because the producer on the server side is not closed.
Failed to create producer: {"errorMsg":"org.apache.pulsar.broker.service.BrokerServiceException$NamingException: Producer with name 'persistent://pulsar/default/input_test-partition-5-0' is already connected to topic","reqId":1766584259806202457, "remote":"21.21.47.12/21.21.47.12:6650", "local":"/9.165.174.197:46786"}
After this, the function restarts due to an exception, and due to failover, the function once again transfers the subscription and fails due to failure to create the producer. Causes the all function pods to constantly restart
Therefore, when I need to enable effectively_once, I have to deploy multiple functions to consume partitioned topics separately. But this is not an easy way to maintain
- In addition, the function can easily fall into the following error and be stuck because of this error until the broker is restarted.
WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x24fe09d6, L:/9.165.182.50:36944 ! R:21.21.134.241/21.21.134.241:6650] Received send error from server: PersistenceError : Cannot determine whether the message is a duplicate at this time
If we have any optimization suggestions, I hope can provide them, thank very much
- Dominant language
- Go
- Stars
- 228
- Forks
- 30
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from streamnative/function-mesh
-
streamnative/function-mesh#811 · 1 assignee ·
-
Difficulty 2/5 Half a day Newbie friendliness 45/100
streamnative/function-mesh#806 ·
-
streamnative/function-mesh#801 · 1 assignee ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
streamnative/function-mesh#796 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
streamnative/function-mesh#769 ·
All issues in streamnative/function-mesh
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
milvus-io/birdwatcher#545 ·
-
kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/prow#953 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
caddyserver/caddy#8046 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
jaegertracing/jaeger#9588 ·