dromara / dromara/hmily

Grpc thread suspend middle and participant doesn't be saved.

Open
#321 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.2k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug
I have a load test with hmily grpc using the demo and found a bug. When I run demo and hit with many concurrent requests, some requests have `hmilyParticipantList is empty`.
After debug, I found something. Thread-0 receive event from grpc, it handle the `GrpcHmilyServerFilter.interceptCall` complete. Nevertheless, after that it handle nothing. After a interval, another thread (not thread-0) handle this event. `ParticipantHmilyTccTransactionHandler` class has `HmilyParticipantCacheManager.getInstance().get(context.getParticipantId())` is empty . It doesn't found hmilyParticipantList in `HmilyTccTransactionExecutor.participantConfirm` step.
I think grpc stop current thread. After that it take another handle same event. Because participant info doesn't be saved before another thread run, the bug raise.

My issue maybe relate with [issue-296](https://github.com/dromara/hmily/issues/296).

## Environment

- hmily version(s): [e.g. 2.1.2-SNAPSHOT]

## Steps to reproduce
Run the demo with a lot concurrent requests.
I run 8 concurrent threads with 5 loops in jmeter.

## Expected behavior
A thread must handle to last and not suspend middle.

## Debug logs
I have some logs, but I add something to debug, so it have a little different with original.

THREAD-0
INFO o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():63 metadata={"transId":-9171302586210725888,"participantId":-9171302566614937600,"action":1,"role":1,"transType":"TCC","spanContext":"{\"hmily_traceid\":\"0c687775d034a36a:95fd1c59263f9d5e:8fc82a36473ce73e:1\"}"}
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.countGrpcRequest():89 start count grpc req step
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():65 done count grpc req step

THREAD-1
INFO o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():63 metadata={"transId":-9171302586210725888,"participantId":-9171302566614937600,"action":2,"role":1,"transType":"TCC","spanContext":"{\"hmily_traceid\":\"0c687775d034a36a:07b222f172063b69:6c288a5be7db75c6:1\"}"}
INFO o.d.h.t.h.ParticipantHmilyTccTransactionHandler.handleTransaction():62 handleTransaction, context=HmilyTransactionContext(transId=-9171302586210725888, participantId=-9171302566614937600, participantRefId=null, action=2, role=1, transType=TCC, xaParticipant=null, spanContext={"hmily_traceid":"0c687775d034a36a:07b222f172063b69:6c288a5be7db75c6:1"})
INFO o.d.h.t.h.ParticipantHmilyTccTransactionHandler.handleTransaction():97 Start CONFIRMING
INFO o.d.h.t.e.HmilyTccTransactionExecutor.participantConfirm():187 Start participantConfirm()
INFO o.d.h.t.e.HmilyTccTransactionExecutor.participantConfirm():190 hmilyParticipantList is empty
ERROR o.d.h.d.g.a.s.AccountServiceGrpcImpl.payment():54 payment() occurs exception,
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.close():77 ForwardingServerCall close status Status{code=UNKNOWN, description=null, cause=java.lang.NullPointerException
INFO o.d.h.g.f.GrpcHmilyServerFilter.close():79 method=AccountService/payment, status=UNKNOWN, responseTime=10

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with GrpcHmilyServerFilter.interceptCall and close, then trace ParticipantHmilyTccTransactionHandler.handleTransaction into HmilyTccTransactionExecutor.participantConfirm and HmilyParticipantCacheManager. Reproduce the demo under load with JMeter using 8 concurrent threads and 5 loops. Done means the participant list is retained and the request no longer fails with an empty list or NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.