elastic / elastic/apm-agent-java

Quartz job instrumentation - distributed tracing pass through JobDataMap

Open
#3,050 6 comments 1 reaction 0 assignees View on GitHub
agent-java community enhancement
Dominant language
Java
Stars
594
Forks
338
Avg merge
1d 13h
Merged PRs (30d)
25

Description

## Is your feature request related to a problem?
We have some Quartz jobs immediately triggered by user requests. Both (job and user request) create transaction in APM, with no link between them: user request appears in "request" transactions, Quartz job appears in "scheduled" transaction, as per design I guess.
We would like to keep trace between user request and the triggered job (when jobs are immediately triggered, maybe not when they're scheduled).

## Describe the solution you'd like
I think Quartz Job APM plugin could retrieve headers from JobDetailMap if any, thus keeping trace between parent and child transactions.

It could be done there:

https://github.com/elastic/apm-agent-java/blob/9c4e9351dd39b4413b2fb4990a73d91674201838/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/AbstractJobTransactionNameInstrumentation.java#L117

We could start a child transaction with few modifications:

AbstractJobTransactionNameInstrumentation line 84:
```java
if (transactionName != null) {
transaction = createAndActivateTransaction(clazz, transactionName, jobExecutionContext, helper);
} else {
logger.warn("Cannot correctly name transaction for method {} because JobExecutionContext is null or lacking job details", signature);
transaction = createAndActivateTransaction(clazz, signature, jobExecutionContext, helper);
}
```

AbstractJobTransactionNameInstrumentation.java line 115:

```java
@Nullable
private static Transaction createAndActivateTransaction(Class originClass, String name, T jobExecutionContext, JobExecutionContextHandler helper) {
Transaction transaction = GlobalTracer.get().startChildTransaction(helper.getJobDataMap(jobExecutionContext), key -> dataMap.getString(key), PrivilegedActionUtils.getClassLoader(originClass));
// ....
```

Add getJobDataMap to https://github.com/elastic/apm-agent-java/blob/main/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/JobExecutionContextHandler.java:

```java
@Nullable
JobDataMap getJobDataMap(T jobExecutionContext);
```

## Describe alternatives you've considered

I've considered implementing it with APM Agent API like this:

Job scheduling:
```java
Span span = ElasticApm.currentSpan();
if (span != null) {
span.injectTraceHeaders((name, value) -> jobDetail.getJobDataMap().put(name, value));
}
/*
this.scheduler.scheduleJob(jobDetail, trigger);
*/
```

Job execution:

```java
Transaction transaction = ElasticApm.startTransactionWithRemoteParent((key) -> jobExecutionContext.getJobDetail().getJobDataMap().getString(key));
try (final Scope scope = transaction.activate()) {
customRunJob(jobExecutionContext);
} catch (Exception e) {
transaction.captureException(e);
throw e;
} finally {
transaction.end();
}
```

It does create a transaction, but interrupt transaction created by apm-quartz-job-plugin. I lose some spans (executed before I start the transaction), which appear in the plugin transaction. I think implementing distributed tracing between executor and job execution at plugin level would be more accurate.

## Additional context

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.