Combining OpenTelemetry and Java auto-instrumentation

TIL you can use the Java OpenTelemetry APIs alone - without setting up an export pipeline - and let auto-instrumentation collect the telemetry for you.

As a developer you want all of your downstream calls to be automatically instrumented - HTTP clients, SQL clients and so on - which auto-instrumentation is great at!

But, you also want to be able to customize spans and attributes manually at will, like this:

@WithSpan("doImportantWork")
private static void importantWork() {

  // Add span attributes
  Span currentSpan = Span.current();
  currentSpan.setAttribute("work.type", "important");
  currentSpan.setAttribute("moon.phase", "waxing gibbous");
}

Explicit OTel Configuration

However to do this explicitly by talking OTLP at some OTLP endpoint you need all this other stuff - a pile of dependencies:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.36.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.36.0</version>
</dependency>
<!-- a pile of other deps omitted for brevity; you get the point! -->

… as well as the exporter configuration in code-land; thanks to the SDK autoconfigure, this isn’t too complex:

package otel;

import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class AutoConfiguredSdk {
  public static OpenTelemetrySdk autoconfiguredSdk() {
    return AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();
  }
}

You’ll also need to setup a handful of environment variables to tell OTel how to autoconfigure itself - what your service is called, its version, how to connect to the OTLP server, and so on.

This is all fine I guess, but it’s a lot of ceremony and dependencies to do something pretty boilerplate.

Auto-instrumentation Configuration

When you auto-instrument, you launch your application’s using the -javaagent flag pointing at your auto-instrumentation tooling - for instance, Datadog - which uses Java’s instrumentation interface to weave instrumentation around things the instrumentation tooling supports - for instance, your HTTP clients, SQL clients, and so on.

It turns out that - and this was a surprise to me - it will also do this for OpenTelemetry API usage when the exporters have not been configured. So we can take our code sample above showing how we mark up void importantWork() using the OTel API, take a single dependency on opentelemetry-api in our POM:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.36.0</version>
</dependency>
<!-- that's it folks -->

… wrap our app at runtime with auto-instrumentation via -javaagent and it will do exactly what we expect, without all the toil! If we’re using Datadog to do our auto-instrumentation, we can even use tooling like the Datadog serverless CDK construct and Kubernetes auto-instrumentation admission controller to make the runtime configuration part implicit.

Passing Thoughts

This feels like a real sweet spot to me; you get the benefits of auto-instrumentation of all your deps, and you can push custom telemetry out on your spans without having to configure a full OTLP pipeline.