awslabs / awslabs/disco

DiSCo(Concurrency) unable to propagate context in ForkJoinTask

Open
#12 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
70
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Following the guidelines provided in https://docs.aws.amazon.com/xray/latest/devguide/aws-x-ray-auto-instrumentation-agent-for-java.html , my hopes were that I could collect tracing data from multiple Java services I have implemented as part of my stack.

My first service is a tool we rely on for CQRS called Axon (https://axoniq.io/download). In our case, we use the AxonServer enterprise (paid), but I was able to replicate the exact same issue on a single node on AxonServer standard edition (open-source). I decided to build my own container and publish to ECR. To package and configure the pre-built JAR file for AxonServer, I created the following Dockerfile:

```
FROM busybox as source

RUN addgroup -S -g 1001 axonserver \
&& adduser -S -u 1001 -G axonserver -h /axonserver -D axonserver \
&& mkdir -p /axonserver/disco/disco-plugins \
&& mkdir -p /axonserver/config /axonserver/data /axonserver/events /axonserver/log \
&& chown -R axonserver:axonserver /axonserver

FROM amazoncorretto/amazoncorretto:15-alpine-full

COPY --from=source /etc/passwd /etc/group /etc/
COPY --from=source --chown=axonserver /axonserver /axonserver

COPY --chown=axonserver axonserver.jar axonserver.properties /axonserver/
COPY --chown=axonserver disco/* /axonserver/disco/

USER axonserver
WORKDIR /axonserver

ENV JAVA_OPTIONS=""
ENV LANG=C.UTF-8

VOLUME [ "/axonserver/config", "/axonserver/data", "/axonserver/events", "/axonserver/log" ]
EXPOSE 8024/tcp 8124/tcp 8224/tcp

ENTRYPOINT java -Duser.timezone=UTC -Djava.security.egd=file:/dev/./urandom -Dlogging.level.com.amazonaws.xray=DEBUG -javaagent:/axonserver/disco/disco-java-agent.jar=pluginPath=/axonserver/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose $JAVA_OPTIONS -jar axonserver.jar
```

Axon exposes 3 ports that does the following: 8024 provides the GUI (and also responsible for readiness and liveness), 8124 is the gRPC responsible to receive commands/queries and dispatch to corresponding handlers, and finally to relay the resulting events to any event handler that may be interested, and finally port 8224 that does cross-node communication for clustering.

My setup is currently an EKS cluster with various ties to AWS services, such as Cloudwatch, Container Insights, ECR, IAM-Authenticator, ALB Ingress Controller, etc. To leverage the integration with X-Ray, I've deployed the X-Ray daemon as a Daemonset in my cluster and exposed them as a headless service reachable through `aws-xray-daemon.aws-system.svc.cluster.local:2000`. Performing a `netcat` successfully connects to the UDP port, meaning daemon is reachable for the containers. Please let me know if you require me to provide the Kubernetes manifests for this deployment. It's inspired on https://www.eksworkshop.com/intermediate/245_x-ray/x-ray-daemon/ with my specific setup/configuration.

My xray daemon contains the following configuration:

```
# Maximum buffer size in MB (minimum 3). Choose 0 to use 1% of host memory.
TotalBufferSizeMB: 0
# Maximum number of concurrent calls to AWS X-Ray to upload segment documents.
Concurrency: 8
# Send segments to AWS X-Ray service in a specific region
Region: "us-east-1"
# Change the X-Ray service endpoint to which the daemon sends segment documents.
Endpoint: ""
Socket:
# Change the address and port on which the daemon listens for UDP packets containing segment documents.
# Make sure we listen on all IP's by default for the k8s setup
UDPAddress: 0.0.0.0:2000
Logging:
LogRotation: true
# Change the log level, from most verbose to least: dev, debug, info, warn, error, prod (default).
LogLevel: prod
# Output logs to the specified file path.
LogPath: ""
# Turn on local mode to skip EC2 instance metadata check.
LocalMode: false
# Amazon Resource Name (ARN) of the AWS resource running the daemon.
ResourceARN: ""
# Assume an IAM role to upload segments to a different account.
RoleARN: ""
# Disable TLS certificate verification.
NoVerifySSL: false
# Upload segments to AWS X-Ray through a proxy.
ProxyAddress: ""
# Daemon configuration file format version.
Version: 2
```

When deploying it on my EKS cluster, the application is fully functional/operational. No traces show up in the AWS Console UI. However, when I turn on the logs, I experience these from disco:

```
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) finished parsing argument list: pluginPath=/axonserver/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinTaskInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinPoolInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ExecutorInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadSubclassInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ScheduledThreadPoolExecutorInterceptor to process
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinPoolInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ExecutorInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadSubclassInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ScheduledThreadPoolExecutorInterceptor
axonserver-0 axonserver [software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) agent startup complete

... Axon startup logs here ...

axonserver-0 axonserver [software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor] DiSCo(Concurrency) unable to propagate context in ForkJoinTask
axonserver-0 axonserver [software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor] DiSCo(Concurrency) unable to propagate context in ForkJoinTask

... Same error keeps repeating undefinitely ...
```

Please let me know if there is anything else I can provide or do to help you troubleshoot this error.

Contributor guide

Open the contributing guide

Research direction

Start with software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor and reproduce the repeated warning using the supplied AxonServer Java agent setup. Verify how ForkJoinTask context propagation behaves, with completion indicated by the warning stopping and AWS X-Ray traces appearing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, java, kubernetes
Domain
devtools, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.