apache / apache/ignite

ignite 2 nodes goes down when i start spark job.

Open
#10,554 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

i have six node ignite cluster and whenever i start the spark streaming job, 2 ignite goes down with below error.

[2023-02-21 14:09:13,968][WARN ][grid-timeout-worker-#214][PoolProcessor] >>> Possible starvation in striped pool.
Thread name: sys-stripe-13-#14
Queue: []
Deadlock: false
Completed: 104
Thread [name="sys-stripe-13-#14", id=30, state=WAITING, blockCnt=506, waitCnt=101]
Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5246d2a6, ownerName=exchange-worker-#283, ownerId=327]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

[2023-02-21 14:10:02,814][ERROR][tcp-disco-msg-worker-[a37cbeb9 192.168.2.7:47500]-#2-#277][root] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-31, igniteInstanceName=null, finished=false, heartbeatTs=1676968733815]]]

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the six-node Ignite cluster failure when the Spark streaming job begins, then inspect the supplied striped-pool starvation and SYSTEM_WORKER_BLOCKED traces. Done means identifying why two nodes halt and documenting or validating a fix that prevents the nodes from going down during the Spark workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.