apache / apache/texera

Akka worker timeout on a specific case

Open
#3,560 0 comments 0 reactions 0 assignees View on GitHub
low-priority
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

## Issue
We get Akka timeout in workflow execution in a specific case.

## Symptoms
Currently, the reason is not clear but the workflow and the dataset is uploaded.

## Error message
```
[2025-07-12 00:20:58,717] [ERROR] [CONTROLLER] [AsyncRPCServer] [Amber-akka.actor.default-dispatcher-14] - Exception occurred
java.util.concurrent.TimeoutException: Future timed out after [5 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait0(Promise.scala:248)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:261)
at scala.concurrent.Await$.$anonfun$result$1(package.scala:201)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:174)
at java.base/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:172)
at scala.concurrent.Await$.result(package.scala:124)
at edu.uci.ics.amber.engine.architecture.common.WorkflowActor.$anonfun$new$1(WorkflowActor.scala:94)
at edu.uci.ics.amber.engine.architecture.common.AkkaActorService.getClusterNodeAddresses(AkkaActorService.scala:42)
at edu.uci.ics.amber.engine.architecture.common.ExecutorDeployment$.createWorkers(ExecutorDeployment.scala:49)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.buildOperator(RegionExecutionCoordinator.scala:267)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.$anonfun$initRegionExecution$1(RegionExecutionCoordinator.scala:249)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.$anonfun$initRegionExecution$1$adapted(RegionExecutionCoordinator.scala:232)
at scala.collection.immutable.BitmapIndexedSetNode.foreach(HashSet.scala:958)
at scala.collection.immutable.HashSet.foreach(HashSet.scala:965)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.initRegionExecution(RegionExecutionCoordinator.scala:232)
at edu.uci.ics.amber.engine.architecture.scheduling.RegionExecutionCoordinator.(RegionExecutionCoordinator.scala:89)
at edu.uci.ics.amber.engine.architecture.scheduling.WorkflowExecutionCoordinator.$anonfun$coordinateRegionExecutors$5(WorkflowExecutionCoordinator.scala:85)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
at scala.collection.immutable.Set$Set1.map(Set.scala:165)
at edu.uci.ics.amber.engine.architecture.scheduling.WorkflowExecutionCoordinator.coordinateRegionExecutors(WorkflowExecutionCoordinator.scala:78)
at edu.uci.ics.amber.engine.architecture.controller.promisehandlers.StartWorkflowHandler.startWorkflow(StartWorkflowHandler.scala:42)
at edu.uci.ics.amber.engine.architecture.controller.promisehandlers.StartWorkflowHandler.startWorkflow$(StartWorkflowHandler.scala:36)
at edu.uci.ics.amber.engine.architecture.controller.ControllerAsyncRPCHandlerInitializer.startWorkflow(ControllerAsyncRPCHandlerInitializer.scala:30)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at edu.uci.ics.amber.engine.common.rpc.AsyncRPCServer.invokeMethod(AsyncRPCServer.scala:82)
at edu.uci.ics.amber.engine.common.rpc.AsyncRPCServer.receive(AsyncRPCServer.scala:66)
at edu.uci.ics.amber.engine.architecture.common.AmberProcessor.processDCM(AmberProcessor.scala:72)
at edu.uci.ics.amber.engine.architecture.controller.Controller.$anonfun$processMessages$2(Controller.scala:144)
at edu.uci.ics.amber.engine.architecture.logreplay.ReplayLogManager.withFaultTolerant(ReplayLogManager.scala:77)
at edu.uci.ics.amber.engine.architecture.logreplay.ReplayLogManager.withFaultTolerant$(ReplayLogManager.scala:71)
at edu.uci.ics.amber.engine.architecture.logreplay.EmptyReplayLogManagerImpl.withFaultTolerant(ReplayLogManager.scala:87)
at edu.uci.ics.amber.engine.architecture.controller.Controller.processMessages(Controller.scala:143)
at edu.uci.ics.amber.engine.architecture.controller.Controller$$anonfun$handleDirectInvocation$1.applyOrElse(Controller.scala:168)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at edu.uci.ics.amber.engine.architecture.common.WorkflowActor.aroundReceive(WorkflowActor.scala:78)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
```

## Materials

[team v2.json](https://github.com/user-attachments/files/21191646/team.v2.json)

[roblox_games_data.csv](https://github.com/user-attachments/files/21191654/roblox_games_data.csv)

Contributor guide

Open the contributing guide

Research direction

Start with the stack-trace locations in WorkflowActor.scala:94, AkkaActorService.scala:42, and ExecutorDeployment.scala:49, then reproduce the timeout using the uploaded workflow and dataset. Trace why cluster-node lookup or worker creation exceeds five seconds; done means the specific workflow starts without this timeout and the failure is covered by verification where the project provides it.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.