Quartz job instrumentation - distributed tracing pass through JobDataMap

Is your feature request related to a problem?

We have some Quartz jobs immediately triggered by user requests. Both (job and user request) create transaction in APM, with no link between them: user request appears in "request" transactions, Quartz job appears in "scheduled" transaction, as per design I guess.
We would like to keep trace between user request and the triggered job (when jobs are immediately triggered, maybe not when they're scheduled).

Describe the solution you'd like

I think Quartz Job APM plugin could retrieve headers from JobDetailMap if any, thus keeping trace between parent and child transactions.

It could be done there:

https://github.com/elastic/apm-agent-java/blob/9c4e9351dd39b4413b2fb4990a73d91674201838/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/AbstractJobTransactionNameInstrumentation.java#L117

We could start a child transaction with few modifications:

AbstractJobTransactionNameInstrumentation line 84:

if (transactionName != null) {
    transaction = createAndActivateTransaction(clazz, transactionName, jobExecutionContext, helper);
} else {
    logger.warn("Cannot correctly name transaction for method {} because JobExecutionContext is null or lacking job details", signature);
    transaction = createAndActivateTransaction(clazz, signature, jobExecutionContext, helper);
}

AbstractJobTransactionNameInstrumentation.java line 115:

@Nullable
private static <T> Transaction createAndActivateTransaction(Class<?> originClass, String name, T jobExecutionContext, JobExecutionContextHandler<T> helper) {
    Transaction transaction = GlobalTracer.get().startChildTransaction(helper.getJobDataMap(jobExecutionContext), key -> dataMap.getString(key), PrivilegedActionUtils.getClassLoader(originClass));
// ....

Add getJobDataMap to https://github.com/elastic/apm-agent-java/blob/main/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/JobExecutionContextHandler.java:

@Nullable
JobDataMap getJobDataMap(T jobExecutionContext);

Describe alternatives you've considered

I've considered implementing it with APM Agent API like this:

Job scheduling:

Span span = ElasticApm.currentSpan();
if (span != null) {
  span.injectTraceHeaders((name, value) -> jobDetail.getJobDataMap().put(name, value));
}
/*
 this.scheduler.scheduleJob(jobDetail, trigger);
*/

Job execution:

Transaction transaction = ElasticApm.startTransactionWithRemoteParent((key) -> jobExecutionContext.getJobDetail().getJobDataMap().getString(key));
try (final Scope scope = transaction.activate()) {
  customRunJob(jobExecutionContext);
} catch (Exception e) {
  transaction.captureException(e);
  throw e;
} finally {
  transaction.end();
}

It does create a transaction, but interrupt transaction created by apm-quartz-job-plugin. I lose some spans (executed before I start the transaction), which appear in the plugin transaction. I think implementing distributed tracing between executor and job execution at plugin level would be more accurate.

Additional context

Inizia con AbstractJobTransactionNameInstrumentation.java intorno alle righe 84 e 115, quindi esamina JobExecutionContextHandler.java. Traccia come viene ottenuto JobDataMap per l’esecuzione dei job Quartz e come viene creata la transazione esistente. Il lavoro è completato quando il plugin può usare gli header di trace da JobDetailMap per preservare il collegamento della transazione padre-figlio senza creare una seconda transazione o perdere gli span esistenti.