graphql-java, Virtual Threads and a new engine mechanism

Commencez par lire graphql.execution.ExecutionStrategy, InstrumentationContext, DataFetcherExceptionHandler et GraphQL.Builder afin de comprendre comment les choix de moteur basés sur CompletableFuture atteignent l’API publique. L’issue présente plusieurs conceptions de Virtual Thread et de moteur, mais n’identifie ni une approche retenue, ni des fichiers d’implémentation, ni des tests, ni une Definition of Done.

A treatise on Virtual Threads and how the might apply to graphql-java.

graphql-java and Virtual Threads

JDK 21 is bringing in the long awaited virtual threads (hereafter VTs). They allow for a more natural way to return asynchronous values.

If a virtual thread performs a blocking IO operation, the VT is suspended and moved off the carrier thread and that carrier thread can be used for other purposes.

This of course has implications for graphql-java.

Todays graphql engine

Today the graphql-java engine is powered by CompleteableFutures (hereafter CFs) - every value returned for a graphql field is encapsulated into a CF and composed together to make the final Map<String,Object> that is placed into the ExecutionResult data element.

So given a query like this

query q {
    films {
        name
        releaseDate
        director {
            name
            born
        }
        cast {
            name
            relatedFilms {
                name
            }
        }

    }
}

the 10 fields (and associated lists) would be wrapped in CFs, and composed down to a final result.

The graphql engine itself today NEVER blocks because its uses CFs for everything. It also never starts a thread today, but rather relies on the outside calling framework to control threading.

If a field fetch computation, ie done by the DataFetcher, wants do perform a blocking IO operation it is responsible for setting up the thread code to do that. It can return a CF like this to ensure the engine running the query starts asynchronous.

DataFetcher dataFetcher = dfe -> {
    return CompleteableFuture.supplyAsync(() -> someCodeThatMayBlock(dfe), executor)
}

So the graphql engine never starts a thread but rather the callback code is responsible for that.

If a DataFecther returns a CF, it will be used. Any other value is wrapped in a CF and then passed back into the engine processing.

So far CFs have proven to be quite fast. While there is some memory pressure involved in the wrapping sync values with CFs, its not overwhelming and the https://en.wikipedia.org/wiki/Treiber_stack based mechanism
in CF has proven to be fast.

In fact some early testing from others like Quarkus have show that in a request / response situation, CFs are nominally marginally faster than the new VTs. Not by much but they certainly are just as fast if not faster.

All benchmarks are lies so don't read too much into other than to say VTs will NOT be significantly faster than CFs.

With virtual threads

The graphql engine could in theory be simplified by embracing virtual threads. Instead of a DataFetcher having to returns a CF to be async, it could itself start a VT and just return an object.

This means that the engine would not need to compose together CFS to get values. A list of N fields would just be a materialised list of N values and they could be returned internally as such.

In theory this could be virally done in all code right ot the stop where CompletableFuture<ExecutionResult> executeAsync(ExecutionInput executionInput) becomes ExecutionResult executeAsync(ExecutionInput executionInput)

A DataFetcher in a VT world could be the following

DataFetcher dataFetcher = dfe -> {
   try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
        Supplier<Object>  value  = scope.fork(() -> someCodeThatMayBlock(dfe));
        scope.join()           
             .throwIfFailed();  // ... and propagate errors
        return value.get()
    }}

Note the above uses the proposed Structured Concurrency API, also in JDK 21 but in preview.

This is actual a return to the past. The original java 6 based engine worked in this manner where all returned values appear synchronous.

The every DataFetcher in a VT idea

In theory we could start a VT for every field fetch. There can be millions of VTs in theory. However VTs are cheap but not free so this does not strike me as a sensible idea.

The non trivial DataFetcher idea

One idea is a that the graphql engine could start a virtual thread per non trivial data fetcher.

Today graphql-java has TrivialDataFetcher which PropertyDataFetcher uses when it reads POJO values out of an in memory object. a VT per field is overkill and not needed.

But a VT per non TrivialDataFetcher might be used so that some one can writing blocking code and the engine running it would stay async and continue to process other fields as it operates.

There are two ways to do this.

The engine itself could start starting VTs itself before executing a non trivial data fetcher.

Or it could provide helper functions that make a DataFetcher start in VT and then it would be a choice of consumer to choose to use it.

  DataFetcher dataFetcher = VirtualThreadDataFetcher.create( dfe -> someCodeThatMayBlock(dfe))

At this stage I really like the fact that graphql-java never starts a thread, virtual or not and leaves it to consumers. That way the threading decisions are left to consumers and frameworks
such as Spring or DGS.

VTs are much much cheaper than old school threads but they are not free or without consequences.

The migration

graphql-java is based on Java 11, having only just updated. We are deliberately slow in base JDK upgrades because all sorts of people use graphql-java and we won't require them to use JDK 21 say for many years.

We want graphql-java to be widely useable on many versions of the JDK.

So we have two cohorts of users. Those who might be able to run JDK 21 and VTs and those who cannot

A CF based engine on JDK21

The current engine can certainly run async in a JDK 21 VT environment. If a DataFetcher did the following it would fit straight into the current engine

DataFetcher dataFetcher = dfe -> {
   try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
        Supplier<Object>  value  = scope.fork(() -> someCodeThatMayBlock(dfe));
        scope.join()           
             .throwIfFailed();  // ... and propagate errors
        return value.get()
    }}

The above would be wrapped in a CF (as it would be completed) by the engine and it would work as it does today.

A object based engine on JDK21

Another option is to rewrite a new engine for those who can exclusively run on JDK 21 and VTs. It would be materialised object based and not use CFs.

DataFetchers would not be affected because the already return Object and we smart wrap them in CFs today and would not need to in the this future engine.

Flow on effects to the engine

This has flow on effects however to other parts of the callback code, namely Instrumentation

Today it has methods like

default InstrumentationContext<Object> beginFieldFetch(InstrumentationFieldFetchParameters parameters, InstrumentationState state) {


public interface InstrumentationContext<T> {

    /**
     * This is invoked when the instrumentation step is initially dispatched
     *
     * @param result the result of the step as a completable future
     */
    void onDispatched(CompletableFuture<T> result);

    /**
     * This is invoked when the instrumentation step is fully completed
     *
     * @param result the result of the step (which may be null)
     * @param t      this exception will be non null if an exception was thrown during the step
     */
    void onCompleted(T result, Throwable t);

}

The CFs here are strongly present in the Instrumentation API. We could leave it as is allocated a dummy CF but thats pointless really if we are writing a new engine

What this means is that a new engine requires a new peer Instrumentation. We also know this to be true from other work we have done where we investigated a new algorithm of fetching and hence the Instrumentation callbacks become specific to the engine algorithm.

Also today the engine used is called an graphql.execution.ExecutionStrategy and its specified when you build the GraphQL instance. These have CFs in their signature like the following (and more)

public abstract CompletableFuture<ExecutionResult> execute(ExecutionContext executionContext, ExecutionStrategyParameters parameters) throws NonNullableFieldWasNullException;

protected CompletableFuture<ExecutionResult> completeValueForObject(ExecutionContext executionContext, ExecutionStrategyParameters parameters, GraphQLObjectType resolvedObjectType, Object result) {

The use of specific ExecutionStrategy and Instrumentation leaks up into high layers via this being used when you build a GraphQL

        GraphQL graphQL = GraphQL.newGraphQL(schema)
                .queryExecutionStrategy(new AsyncExecutionStrategy())
                .instrumentation(new TracingInstrumentation())
                .build();

We need a marker like interface that means that you can build a GraphQL instance but specify the key engine bits in a manner that is not so tied to its shape. eg rather this

public Builder queryExecutionStrategy(ExecutionStrategyMarkerInterface executionStrategy) {


public Builder instrumentation(InstrumentationMarkerInterface instrumentation) {

or better yet invent something like a GraphqlEngine that holds these runtime concerns in it and this is then set into the GraphQL instance. Something like

        GraphQLEngine engine = GraphQLEngine.newEngine()
                .queryStrategy(new AsyncExecutionStrategy())
                .instrumentation(new TracingInstrumentation())
                .build();

        GraphQL graphQL = GraphQL.newGraphQL(schema)
                .engine(engine)
                .build();

This would then allow say

        GraphQLEngine engine = GraphQLEngine.newEngine()
                .queryEngine(new Jdk21VirtualThreadEngine())
                .instrumentation(new Jdk21TracingInstrumentation())
                .build();

or more likely since the engine parts actually need to come together as one part they would be a specific class to construct them

        GraphQLEngine engine = Jdk21VirtualThreadEngine.newEngine().build();
        GraphQL graphQL = GraphQL.newGraphQL(schema)
                .engine(engine)
                .build();

Wrinkles in current code

graphql.execution.preparsed.PreparsedDocumentProvider uses CFs however I think in a future VT world its ok to
expect it to return a completed CF wrapper of the PreparsedDocumentEntry. It's not worth the effort to change this.

DataFetcherExceptionHandler has a CF in its signature via

public interface DataFetcherExceptionHandler {
    CompletableFuture<DataFetcherExceptionHandlerResult> handleException(DataFetcherExceptionHandlerParameters handlerParameters);
}

however its really tied to the engine instance and hence it could be engine specific. The defaulting that happens now via graphql.GraphQL.Builder#defaultDataFetcherExceptionHandler would need to be removed.

The defaulting of graphql.execution.instrumentation.dataloader.DataLoaderDispatcherInstrumentation needs to be taken out of the GraphQL class and moved into the engine factory say.

In Summary

• graphql-java will happily work in a VT world as it is today
• graphql-java has to be able to operate in a pre VT JDK world and a post JDK VT world
• VT support would make for cleaner engine code in that it is values returned not CFs
• There maybe some performance enhancements for simple object values
• There are not likely to be performance enhancements for truly async object values
• A new Engine + Instrumentation mechanism is needed

None of the above is a promise to do anything - but rather an enumeration of ideas.