ignite 2 nodes goes down when i start spark job.
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
i have six node ignite cluster and whenever i start the spark streaming job, 2 ignite goes down with below error.
[2023-02-21 14:09:13,968][WARN ][grid-timeout-worker-#214][PoolProcessor] >>> Possible starvation in striped pool.
Thread name: sys-stripe-13-#14
Queue: []
Deadlock: false
Completed: 104
Thread [name="sys-stripe-13-#14", id=30, state=WAITING, blockCnt=506, waitCnt=101]
Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5246d2a6, ownerName=exchange-worker-#283, ownerId=327]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
[2023-02-21 14:10:02,814][ERROR][tcp-disco-msg-worker-[a37cbeb9 192.168.2.7:47500]-#2-#277][root] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-31, igniteInstanceName=null, finished=false, heartbeatTs=1676968733815]]]
Contributor guide
Research direction
Start by reproducing the six-node Ignite cluster failure when the Spark streaming job begins, then inspect the supplied striped-pool starvation and SYSTEM_WORKER_BLOCKED traces. Done means identifying why two nodes halt and documenting or validating a fix that prevents the nodes from going down during the Spark workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100