Grpc thread suspend middle and participant doesn't be saved.
- Dominant language
- Java
- Stars
- 4.2k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
I have a load test with hmily grpc using the demo and found a bug. When I run demo and hit with many concurrent requests, some requests have `hmilyParticipantList is empty`.
After debug, I found something. Thread-0 receive event from grpc, it handle the `GrpcHmilyServerFilter.interceptCall` complete. Nevertheless, after that it handle nothing. After a interval, another thread (not thread-0) handle this event. `ParticipantHmilyTccTransactionHandler` class has `HmilyParticipantCacheManager.getInstance().get(context.getParticipantId())` is empty . It doesn't found hmilyParticipantList in `HmilyTccTransactionExecutor.participantConfirm` step.
I think grpc stop current thread. After that it take another handle same event. Because participant info doesn't be saved before another thread run, the bug raise.
My issue maybe relate with [issue-296](https://github.com/dromara/hmily/issues/296).
## Environment
- hmily version(s): [e.g. 2.1.2-SNAPSHOT]
## Steps to reproduce
Run the demo with a lot concurrent requests.
I run 8 concurrent threads with 5 loops in jmeter.
## Expected behavior
A thread must handle to last and not suspend middle.
## Debug logs
I have some logs, but I add something to debug, so it have a little different with original.
THREAD-0
INFO o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():63 metadata={"transId":-9171302586210725888,"participantId":-9171302566614937600,"action":1,"role":1,"transType":"TCC","spanContext":"{\"hmily_traceid\":\"0c687775d034a36a:95fd1c59263f9d5e:8fc82a36473ce73e:1\"}"}
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.countGrpcRequest():89 start count grpc req step
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():65 done count grpc req step
THREAD-1
INFO o.d.h.g.f.GrpcHmilyServerFilter.interceptCall():63 metadata={"transId":-9171302586210725888,"participantId":-9171302566614937600,"action":2,"role":1,"transType":"TCC","spanContext":"{\"hmily_traceid\":\"0c687775d034a36a:07b222f172063b69:6c288a5be7db75c6:1\"}"}
INFO o.d.h.t.h.ParticipantHmilyTccTransactionHandler.handleTransaction():62 handleTransaction, context=HmilyTransactionContext(transId=-9171302586210725888, participantId=-9171302566614937600, participantRefId=null, action=2, role=1, transType=TCC, xaParticipant=null, spanContext={"hmily_traceid":"0c687775d034a36a:07b222f172063b69:6c288a5be7db75c6:1"})
INFO o.d.h.t.h.ParticipantHmilyTccTransactionHandler.handleTransaction():97 Start CONFIRMING
INFO o.d.h.t.e.HmilyTccTransactionExecutor.participantConfirm():187 Start participantConfirm()
INFO o.d.h.t.e.HmilyTccTransactionExecutor.participantConfirm():190 hmilyParticipantList is empty
ERROR o.d.h.d.g.a.s.AccountServiceGrpcImpl.payment():54 payment() occurs exception,
DEBUG o.d.h.g.f.GrpcHmilyServerFilter.close():77 ForwardingServerCall close status Status{code=UNKNOWN, description=null, cause=java.lang.NullPointerException
INFO o.d.h.g.f.GrpcHmilyServerFilter.close():79 method=AccountService/payment, status=UNKNOWN, responseTime=10
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with GrpcHmilyServerFilter.interceptCall and close, then trace ParticipantHmilyTccTransactionHandler.handleTransaction into HmilyTccTransactionExecutor.participantConfirm and HmilyParticipantCacheManager. Reproduce the demo under load with JMeter using 8 concurrent threads and 5 loops. Done means the participant list is retained and the request no longer fails with an empty list or NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100