OpenTelemetry for Go

OpenTelemetry for Go

Honeycomb has an OpenTelemetry Distribution for Go that wraps the official OpenTelemetry Go SDK. It simplifies the setup process and lets you add instrumentation to your applications and send telemetry data to Honeycomb.

Requirements 

These instructions will explain how to set up automatic and manual instrumentation for a service written in Go. In order to follow along, you will need:

• Go 1.19 or higher
• An application written in Go
• A Honeycomb API Key. You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Examples 

There is an example that configures an application to send OpenTelemetry data to Honeycomb.

Automatic Instrumentation 

This section describes using instrumentation libraries with the OpenTelemetry Go SDK. For automatic instrumentation using eBPF, refer to Automatic Instrumentation using eBPF.

Acquire Dependencies 

Install the Honeycomb OpenTelemetry Go Distribution package:

go get github.com/honeycombio/honeycomb-opentelemetry-go \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Initialize 

Prepare your application to send spans to Honeycomb.

main.go

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/honeycombio/honeycomb-opentelemetry-go"
    "github.com/honeycombio/otel-config-go/otelconfig"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

// Implement an HTTP Handler function to be instrumented
func httpHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

// Wrap the HTTP handler function with OTel HTTP instrumentation
func wrapHandler() {
    handler := http.HandlerFunc(httpHandler)
    wrappedHandler := otelhttp.NewHandler(handler, "hello")
    http.Handle("/hello", wrappedHandler)
}

func main() {
    // Enable multi-span attributes
    bsp := honeycomb.NewBaggageSpanProcessor()

    // Use the Honeycomb distro to set up the OpenTelemetry SDK
    otelShutdown, err := otelconfig.ConfigureOpenTelemetry(
        otelconfig.WithSpanProcessor(bsp),
    )
    if err != nil {
        log.Fatalf("error setting up OTel SDK - %e", err)
    }
    defer otelShutdown()

    // Initialize HTTP handler instrumentation
    wrapHandler()
    log.Fatal(http.ListenAndServe(":3030", nil))
}

Configure 

Use environment variables to configure the Honeycomb OpenTelemetry Go distribution package:

export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"

OTEL_SERVICE_NAMEHONEYCOMB_API_KEY

export HONEYCOMB_DATASET="your-dataset"

Advanced Configuration 

This is the complete list of configuration options for the Honeycomb Distribution.

HONEYCOMB_API_KEYOTEL_SERVICE_NAMEunknown_serviceservice.nameHONEYCOMB_TRACES_APIKEYHONEYCOMB_API_KEYHONEYCOMB_METRICS_APIKEYHONEYCOMB_API_KEYHONEYCOMB_METRICS_DATASETHONEYCOMB_API_ENDPOINThttps://api.honeycomb.io:443HONEYCOMB_TRACES_API_ENDPOINTHONEYCOMB_API_ENDPOINTHONEYCOMB_API_ENDPOINTHONEYCOMB_METRICS_API_ENDPOINTHONEYCOMB_API_ENDPOINTHONEYCOMB_API_ENDPOINTSAMPLE_RATE1OTEL_METRICS_ENABLEDfalseHONEYCOMB_ENABLE_LOCAL_VISUALIZATIONSfalseDEBUGfalse

Run 

Run your application:

go run YOUR_APPLICATION_NAME.go

YOUR_APPLICATION_NAME

In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.

Using HTTP/protobuf instead of gRPC 

OTEL_EXPORTER_OTLP_PROTOCOL

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

Adding Manual Instrumentation 

Acquiring a Tracer 

Tracer

import (
  // ...
  "go.opentelemetry.io/otel"
  // ...
)

// ...
tracer := otel.Tracer("tracer.name.here")

Tracer

Tracerlibrary.name

In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.

library.name

Adding Attributes to Spans 

net/http

import (
  // ...
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
  // ...
)

// ...
handler := func(w http.ResponseWriter, r *http.Request) {
  user := someServiceCall() // get the currently logged in user
  ctx := r.Context()
  span := trace.SpanFromContext(ctx)
  span.SetAttributes(attribute.Int("user.id", user.getID()))
}
// ...

user.idWHEREGROUP BYORDER

Creating Spans 

Instrumentation libraries can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what is happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:

import (
  // ...
  "go.opentelemetry.io/otel"
  // ...
)

// ...
tracer := otel.Tracer("my-app") // if not already in scope
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
// ...

Multi-Span Attributes 

Sometimes you will want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.

keyvalueBaggageSpanProcessor

The example below shows how you can add a user ID attribute to multiple spans in a trace.

import (
  // ...
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
  "go.opentelemetry.io/otel/baggage"
  // ...
)

// ...
handler := func(w http.ResponseWriter, r *http.Request) {
  user := someServiceCall() // get the currently logged in user
  ctx := r.Context()

  // add the user ID attribute to baggage and create new context
  bag := baggage.FromContext(ctx)
  multiSpanAttribute, _ := baggage.NewMember("user.id", user.getID)
  bag, _ = bag.SetMember(multiSpanAttribute)
  ctx = baggage.ContextWithBaggage(ctx, bag)

  tracer := otel.Tracer("my-app") // if not already in scope

  // every subsequent span created from this context, and any of its child spans,
  // will have the user ID attribute from baggage
  ctx, span := tracer.Start(ctx, "expensive-operation")
  defer span.End()
}
// ...

baggage.NewMember

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

Span Status Enum Differences 

Code

Code

Ok1Error21Ok2Error

More on Manual Instrumentation 

The OpenTelemetry documentation for Go has a comprehensive set of topics on manual instrumentation.

Sampling 

SAMPLE_RATE

export SAMPLE_RATE=2

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.

By default, the Honeycomb’s OpenTelemetry Distribution for Go uses the W3C trace context format.

If you opt to use a different trace context specification than W3C, ensure that both the sending and receiving service are using the same propagation format, and that both services are configured to send data to the same Honeycomb environment.

Visualize Traces Locally 

Honeycomb’s OpenTelemetry Distribution for Go can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation.

HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONStrue

export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true

Then, run your application:

go run main.go

The output displays the name of the root span and a link to Honeycomb that shows its trace. For example:

Trace for root-span-name
Honeycomb link: <link to Honeycomb trace>

Using OpenTelemetry Without the Honeycomb Distribution 

The primary purpose of Honeycomb’s Distribution for Go is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Go, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.

api.honeycomb.io

Follow the directions below to instrument with OpenTelemetry for Go.

Install Packages 

go get \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Configure Environment Variables 

export OTEL_SERVICE_NAME="your-service-name"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"

Run 

package main

import (
  "context"
  "fmt"
  "log"
  "net/http"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
  "go.opentelemetry.io/otel/propagation"
  "go.opentelemetry.io/otel/sdk/trace"

  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
  ctx := context.Background()

  // Configure a new OTLP exporter using environment variables for sending data to Honeycomb over gRPC
  client := otlptracegrpc.NewClient()
  exp, err := otlptrace.New(ctx, client)
  if err != nil {
    log.Fatalf("failed to initialize exporter: %e", err)
  }

  // Create a new tracer provider with a batch span processor and the otlp exporter
  tp := trace.NewTracerProvider(
    trace.WithBatcher(exp),
  )

  // Handle shutdown to ensure all sub processes are closed correctly and telemetry is exported
  defer func() {
    _ = exp.Shutdown(ctx)
    _ = tp.Shutdown(ctx)
  }()

  // Register the global Tracer provider
  otel.SetTracerProvider(tp)

  // Register the W3C trace context and baggage propagators so data is propagated across services/processes
  otel.SetTextMapPropagator(
    propagation.NewCompositeTextMapPropagator(
      propagation.TraceContext{},
      propagation.Baggage{},
    ),
  )

  // Implement an HTTP handler func to be instrumented
  handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
  })

  // Setup handler instrumentation
  wrappedHandler := otelhttp.NewHandler(handler, "hello")
  http.Handle("/hello", wrappedHandler)

  // Start web server
  log.Fatal(http.ListenAndServe(":3030", nil))
}

export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset"

Sampling Without the Honeycomb SDK 

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.

TraceIdRatioBased1/NSampleRateN

In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2"

Endpoint URLs for OTLP/HTTP 

OTEL_EXPORTER_OTLP_ENDPOINT

v1/tracesv1/metrics

OTEL_EXPORTER_OTLP_ENDPOINThttps://api.honeycomb.iohttps://api.honeycomb.io/v1/traceshttps://api.honeycomb.io/v1/metrics

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io

OTEL_EXPORTER_OTLP__ENDPOINTOTEL_EXPORTER_OTLP_ENDPOINT

OTEL_EXPORTER_OTLP_TRACES_ENDPOINTv1/tracesOTEL_EXPORTER_OTLP_METRICS_ENDPOINTv1/metrics

Send both traces and metrics to Honeycomb using this method by setting the following variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics

More details about endpoints and signals can be found in the OpenTelemetry Specification.

Automatic Instrumentation using eBPF 

To instrument http and gRPC requests in Go typically requires wrapping requests with OpenTelemetry instrumentation libraries. There is a new project that allows for automatic instrumentation of http and gRPC requests using eBPF, which requires no application code changes.

Because the automatic instrumentation uses eBPF, it requires a Linux kernel. Automatic instrumentation should work on any Linux kernel above 4.4. For most cloud-native applications, this means including the docker image to run as an agent in a container for each application.

Configure 

There are three main configuration options needed for each instrumented application:

OTEL_GO_AUTO_TARGET_EXEOTEL_EXPORTER_OTLP_ENDPOINTOTEL_SERVICE_NAME

Configure an OpenTelemetry Collector to receive traces over OTLP/gRPC and export those traces to Honeycomb. Then follow the instructions for deployment in Kubernetes or from source on a Linux machine.

Running in Kubernetes 

shareProcessNamespacesecurityContext

spec.template.spec

    spec:
      shareProcessNamespace: true
      securityContext: {}
      terminationGracePeriodSeconds: 30
      containers:
        - name: my-service
          image: my-service:v42
          ports:
          - containerPort: 7007
            name: http
        - name: my-service-instrumentation
          image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.2.0-alpha
          env:
          - name: OTEL_GO_AUTO_TARGET_EXE
            value: /app/my-service
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: http://otel-collector:4317
          - name: OTEL_SERVICE_NAME
            value: my-service-name
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true

Example 

There is an example that illustrates a Kubernetes deployment.

Running on a Linux Machine 

otel-go-instrumentation

Set environment variables for the application, service name, and endpoint, and pass into a run command with the instrumentation.

~/app/my-service

OTEL_GO_AUTO_TARGET_EXE=~/app/my-service \ # application being instrumented
  OTEL_SERVICE_NAME=my-service \ # name of service in telemetry data
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \ # send to collector
  ./otel-go-instrumentation

sudo

Troubleshooting 

No Traces for a Service 

unknown_service

My Traces have Duplicate Spans 

This trace has multiple spans sharing the same non-null span ID

One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.

Printing to the Console 

DEBUGstdoutstdout

export DEBUG=true

Keep in mind that printing to the console should only be used for debugging purposes. It is not recommended for production.

Exporting to an Insecure Endpoint 

By default, the Honeycomb Distribution uses a secure exporter. To export to an insecure endpoint, such as a local collector on the same network, set the Insecure option for the exporter with an environment variable:

export OTEL_EXPORTER_OTLP_INSECURE=true

To set the Insecure option in code instead of an environment variable:

otelconfig.ConfigureOpenTelemetry(otelconfig.WithExporterInsecure(true))

OTLP Protobuf Definitions 

Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP protobuf definitions are 0.7.0 for traces and metrics.

If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.

Receiving 464 Errors 

464