open-telemetry / open-telemetry/opentelemetry-java

OpenJDK CRaC support

Open
#6,756 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request help wanted
Dominant language
Java
Stars
2.5k
Forks
1k
Avg merge
3d 17h
Merged PRs (30d)
58

Description

Is your feature request related to a problem? Please describe.

I have a spring boot app which uses opentelemetry-javaagent. I wanted to utilize CRaC but javaagent is preventing CRaC from capturing the endpoint becuse it keeps files and connections open.

I have posted the issue on javaagent repository, and hoped for the resolution.

Then I started to work on a workaround, that is, removing javaagent completely and utilizing spring-boot-starter along with some intstrumentation libraries. I have used SDK-s autoconfiguration.

I had some progress, but in the end, CRaC still couldn't work with that setup, because the way Opentelemetry SDK is designed it doesn't allow reinitialization of OpenTelemetry bean, and more importantly the exporter which is used can only be shutdown, and not relinitialized again. I even tried reinitializing the exporter, but SdkTracerProvider has no method for replacing current exporter.

Then I abandoned autoconfiguration and I tried to go up the heirarchy of classes and replace every bean in Opentelemetry Configuration, where i eventually wanted to do this:

OpenTelemetrySdk newOpenTelemetrySdk = OpenTelemetrySdk.builder()
            .setTracerProvider(newTracerProvider)
            .buildAndRegisterGlobal();

but this failed since it's not allowed to call .buiildAndRegisterGlobal() more than once in app.

I tried all of this just to close the exporter connection to otel collector, before capturing CRaC checkpoint, and open the connection after the checkpoint is restored.
More info about that here.

After all of this the only option was to manually do everything including exposing my own OpenTelemetry bean and reinitializing it once the checkpoint is restored, but I figured out that in order to do that, and to keep exporting traces i had to reinitialize all the beans which use Opentelemetry which I'm not keen on doing.

For example I'm using mongo instrumentation library which works like charm, but it works by injecting MongoTelemetry bean into MongoClientSettings and then creating MongoClient with those settings. And note that MongoTelemetry depends on OpenTelemetry bean, so if I wanted to change OpenTelemetry bean I had to reinitialize every mongodb connector.
The point is, changing the whole OpenTelemetry bean along with exporters requires reloading all beans that depend on it which is not really convenient.

So the closest to a solution I have gotten is that I have a spring boot app which uses spring boot starter and mongodb instrumentation library with custom configuration where I could shutdown the exporter and capture CRaC checkpoint, but when the checkpoint is restored, the exporter was in 'shutdown' state and did not export any traces.

Describe the solution you'd like
What I'd like is to have a support for CRaC in this SDK. It would be perfect if you could expose Opentelemetry as a CRaC resource and we register it in our app and checkpointing and restoring works fine.

I understand I might be asking too much, so the other approach which would be sufficient is to just allow the SpanExporters to have open() method so they can open the connection to collectors just like they have shutdown() method for shutting them down.

With that much control I could, hopefully, close the connection to exporter, and reopen it once the checkpoint is restored.

Additional context
As you can imagine, here I am asking for a new feature, but I'm also looking for a solution to my problem. I don't know if this is the best place to post all of this, but I hope you can help me out. Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the SpanExporter interface and OpenTelemetrySdk.builder().buildAndRegisterGlobal(), then compare their lifecycle constraints with the CRaC resource guidance linked in the issue. The desired outcome is CRaC checkpoint/restore support for the SDK, or an exporter lifecycle that permits reopening connections after restore without reinitializing dependent beans.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.