DiSCo(Concurrency) unable to propagate context in ForkJoinTask
- Dominant language
- Java
- Stars
- 70
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Following the guidelines provided in https://docs.aws.amazon.com/xray/latest/devguide/aws-x-ray-auto-instrumentation-agent-for-java.html , my hopes were that I could collect tracing data from multiple Java services I have implemented as part of my stack.
My first service is a tool we rely on for CQRS called Axon (https://axoniq.io/download). In our case, we use the AxonServer enterprise (paid), but I was able to replicate the exact same issue on a single node on AxonServer standard edition (open-source). I decided to build my own container and publish to ECR. To package and configure the pre-built JAR file for AxonServer, I created the following Dockerfile:
```
FROM busybox as source
RUN addgroup -S -g 1001 axonserver \
&& adduser -S -u 1001 -G axonserver -h /axonserver -D axonserver \
&& mkdir -p /axonserver/disco/disco-plugins \
&& mkdir -p /axonserver/config /axonserver/data /axonserver/events /axonserver/log \
&& chown -R axonserver:axonserver /axonserver
FROM amazoncorretto/amazoncorretto:15-alpine-full
COPY --from=source /etc/passwd /etc/group /etc/
COPY --from=source --chown=axonserver /axonserver /axonserver
COPY --chown=axonserver axonserver.jar axonserver.properties /axonserver/
COPY --chown=axonserver disco/* /axonserver/disco/
USER axonserver
WORKDIR /axonserver
ENV JAVA_OPTIONS=""
ENV LANG=C.UTF-8
VOLUME [ "/axonserver/config", "/axonserver/data", "/axonserver/events", "/axonserver/log" ]
EXPOSE 8024/tcp 8124/tcp 8224/tcp
ENTRYPOINT java -Duser.timezone=UTC -Djava.security.egd=file:/dev/./urandom -Dlogging.level.com.amazonaws.xray=DEBUG -javaagent:/axonserver/disco/disco-java-agent.jar=pluginPath=/axonserver/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose $JAVA_OPTIONS -jar axonserver.jar
```
Axon exposes 3 ports that does the following: 8024 provides the GUI (and also responsible for readiness and liveness), 8124 is the gRPC responsible to receive commands/queries and dispatch to corresponding handlers, and finally to relay the resulting events to any event handler that may be interested, and finally port 8224 that does cross-node communication for clustering.
My setup is currently an EKS cluster with various ties to AWS services, such as Cloudwatch, Container Insights, ECR, IAM-Authenticator, ALB Ingress Controller, etc. To leverage the integration with X-Ray, I've deployed the X-Ray daemon as a Daemonset in my cluster and exposed them as a headless service reachable through `aws-xray-daemon.aws-system.svc.cluster.local:2000`. Performing a `netcat` successfully connects to the UDP port, meaning daemon is reachable for the containers. Please let me know if you require me to provide the Kubernetes manifests for this deployment. It's inspired on https://www.eksworkshop.com/intermediate/245_x-ray/x-ray-daemon/ with my specific setup/configuration.
My xray daemon contains the following configuration:
```
# Maximum buffer size in MB (minimum 3). Choose 0 to use 1% of host memory.
TotalBufferSizeMB: 0
# Maximum number of concurrent calls to AWS X-Ray to upload segment documents.
Concurrency: 8
# Send segments to AWS X-Ray service in a specific region
Region: "us-east-1"
# Change the X-Ray service endpoint to which the daemon sends segment documents.
Endpoint: ""
Socket:
# Change the address and port on which the daemon listens for UDP packets containing segment documents.
# Make sure we listen on all IP's by default for the k8s setup
UDPAddress: 0.0.0.0:2000
Logging:
LogRotation: true
# Change the log level, from most verbose to least: dev, debug, info, warn, error, prod (default).
LogLevel: prod
# Output logs to the specified file path.
LogPath: ""
# Turn on local mode to skip EC2 instance metadata check.
LocalMode: false
# Amazon Resource Name (ARN) of the AWS resource running the daemon.
ResourceARN: ""
# Assume an IAM role to upload segments to a different account.
RoleARN: ""
# Disable TLS certificate verification.
NoVerifySSL: false
# Upload segments to AWS X-Ray through a proxy.
ProxyAddress: ""
# Daemon configuration file format version.
Version: 2
```
When deploying it on my EKS cluster, the application is fully functional/operational. No traces show up in the AWS Console UI. However, when I turn on the logs, I experience these from disco:
```
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) finished parsing argument list: pluginPath=/axonserver/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinTaskInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinPoolInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ExecutorInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadSubclassInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ScheduledThreadPoolExecutorInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinPoolInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ExecutorInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadSubclassInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ScheduledThreadPoolExecutorInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) agent startup complete
... Axon startup logs here ...
axonserver-0 axonserver [software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor] DiSCo(Concurrency) unable to propagate context in ForkJoinTask
axonserver-0 axonserver [software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor] DiSCo(Concurrency) unable to propagate context in ForkJoinTask
... Same error keeps repeating undefinitely ...
```
Please let me know if there is anything else I can provide or do to help you troubleshoot this error.
Contributor guide
Research direction
Start with software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor and reproduce the repeated warning using the supplied AxonServer Java agent setup. Verify how ForkJoinTask context propagation behaves, with completion indicated by the warning stopping and AWS X-Ray traces appearing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, java, kubernetes
- Domain
- devtools, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100