Akka worker timeout on a specific case
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
## Issue
We get Akka timeout in workflow execution in a specific case.
## Symptoms
Currently, the reason is not clear but the workflow and the dataset is uploaded.
## Error message
```
[2025-07-12 00:20:58,717] [ERROR] [CONTROLLER] [AsyncRPCServer] [Amber-akka.actor.default-dispatcher-14] - Exception occurred
java.util.concurrent.TimeoutException: Future timed out after [5 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait0(Promise.scala:248)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:261)
at scala.concurrent.Await$.$anonfun$result$1(package.scala:201)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:174)
at java.base/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:172)
at scala.concurrent.Await$.result(package.scala:124)
at edu.uci.ics.amber.engine.architecture.common.WorkflowActor.$anonfun$new$1(WorkflowActor.scala:94)
at edu.uci.ics.amber.engine.architecture.common.AkkaActorService.getClusterNodeAddresses(AkkaActorService.scala:42)
at edu.uci.ics.amber.engine.architecture.common.ExecutorDeployment$.createWorkers(ExecutorDeployment.scala:49)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.buildOperator(RegionExecutionCoordinator.scala:267)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.$anonfun$initRegionExecution$1(RegionExecutionCoordinator.scala:249)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.$anonfun$initRegionExecution$1$adapted(RegionExecutionCoordinator.scala:232)
at scala.collection.immutable.BitmapIndexedSetNode.foreach(HashSet.scala:958)
at scala.collection.immutable.HashSet.foreach(HashSet.scala:965)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.initRegionExecution(RegionExecutionCoordinator.scala:232)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.(RegionExecutionCoordinator.scala:89)
at edu.uci.ics.amber.engine.architecture.scheduling.WorkflowExecutionCoordinator.$anonfun$coordinateRegionExecutors$5(WorkflowExecutionCoordinator.scala:85)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
at scala.collection.immutable.Set$Set1.map(Set.scala:165)
at edu.uci.ics.amber.engine.architecture.scheduling.WorkflowExecutionCoordinator.coordinateRegionExecutors(WorkflowExecutionCoordinator.scala:78)
at edu.uci.ics.amber.engine.architecture.controller.promisehandlers.StartWorkflowHandler.startWorkflow(StartWorkflowHandler.scala:42)
at edu.uci.ics.amber.engine.architecture.controller.promisehandlers.StartWorkflowHandler.startWorkflow$(StartWorkflowHandler.scala:36)
at edu.uci.ics.amber.engine.architecture.controller.ControllerAsyncRPCHandlerInitializer.startWorkflow(ControllerAsyncRPCHandlerInitializer.scala:30)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at edu.uci.ics.amber.engine.common.rpc.AsyncRPCServer.invokeMethod(AsyncRPCServer.scala:82)
at edu.uci.ics.amber.engine.common.rpc.AsyncRPCServer.receive(AsyncRPCServer.scala:66)
at edu.uci.ics.amber.engine.architecture.common.AmberProcessor.processDCM(AmberProcessor.scala:72)
at edu.uci.ics.amber.engine.architecture.controller.Controller.$anonfun$processMessages$2(Controller.scala:144)
at edu.uci.ics.amber.engine.architecture.logreplay.ReplayLogManager.withFaultTolerant(ReplayLogManager.scala:77)
at edu.uci.ics.amber.engine.architecture.logreplay.ReplayLogManager.withFaultTolerant$(ReplayLogManager.scala:71)
at edu.uci.ics.amber.engine.architecture.logreplay.EmptyReplayLogManagerImpl.withFaultTolerant(ReplayLogManager.scala:87)
at edu.uci.ics.amber.engine.architecture.controller.Controller.processMessages(Controller.scala:143)
at edu.uci.ics.amber.engine.architecture.controller.Controller$$anonfun$handleDirectInvocation$1.applyOrElse(Controller.scala:168)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at edu.uci.ics.amber.engine.architecture.common.WorkflowActor.aroundReceive(WorkflowActor.scala:78)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
```
## Materials
[team v2.json](https://github.com/user-attachments/files/21191646/team.v2.json)
[roblox_games_data.csv](https://github.com/user-attachments/files/21191654/roblox_games_data.csv)
Contributor guide
Research direction
Start with the stack-trace locations in WorkflowActor.scala:94, AkkaActorService.scala:42, and ExecutorDeployment.scala:49, then reproduce the timeout using the uploaded workflow and dataset. Trace why cluster-node lookup or worker creation exceeds five seconds; done means the specific workflow starts without this timeout and the failure is covered by verification where the project provides it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100