SpanProcessor API - Storing a Span specific state

As part of doing a PoC for JFR for OT I realized that I need to keep a state in the SpanProcessor so I can start the JFR event for the Span and end it when the span is ended. For the Context this is rather straight forward with a ThreadLocal as it is (should) always be started/ended in the same thread, however Spans do not have that guarantee and thus I need to currently have a shared data structure in the processor for handling the state for different spans.

In the current implementation I did a simple ConcurrentHashMap which works, but has the potential for some so contention for applications with many parallel threads and spans.

I did a basic implementation having the onStart function return the onEnd function as a callback, which looks OK for the most part, but does increase allocation in handling and in particular the MultiSpanProcessor, also do not know how to make it work with the Batch/Async processor as the callback wouldn't be returned immediately.

public interface SpanProcessor {
    @Nullable
    Consumer<ReadableSpan> onStart(ReadableSpan span);
    void shutdown();
    void forceFlush();
}

Another option could be to extend the ability to add a SpanProcessor state to the ReadableSpan object. Could be stored in a map and hashed on SpanProcessor instance.

Any other thoughts or ideas @jkwatson @bogdandrutu? Be happy to try and do some test implementations.

Start with the SpanProcessor and ReadableSpan APIs, then inspect MultiSpanProcessor and the Batch/Async processor behavior described in the issue. Compare the callback and span-state alternatives with test implementations, and consider the work complete when the project has an agreed API for processor-specific span state with coverage for span start and end handling.