Go OpenTelemetry tracing

Use the Go SDK when you need application spans, explicit propagation or operations outside the eBPF sensor's coverage. This example serves GET /hello, exports a server span over OTLP/HTTP and writes a JSON log carrying the same trace and span IDs.

Run the example

You need Go 1.25 or later and an Avuru Obs gateway. Download main.go, go.mod and go.sum into the same directory. The example pins OpenTelemetry Go 1.46.0; keep its modules on matching versions.

For a local exercise, forward the gateway in a separate terminal:

kubectl -n avuruobs port-forward svc/avuruobs-gateway 4318:4318

Then, in the downloaded example directory:

export OTEL_SERVICE_NAME=avuru-go-example
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces
go mod download
go run .

If ingest authentication is enforced, configure OTEL_EXPORTER_OTLP_TRACES_HEADERS with Authorization=Bearer YOUR_INGEST_KEY. Use the key of a test project. Inside another Kubernetes namespace, use the fully qualified gateway hostname, such as avuruobs-gateway.avuruobs.svc.cluster.local, rather than localhost.

Make a request from another terminal:

curl --fail http://127.0.0.1:8081/hello
# hello from avuru-go-example

Complete source

main.go

// A minimal OTLP/HTTP tracing example. Run against a test receiver.
package main

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func run() error {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	// Endpoint, protocol and headers use standard OTEL_EXPORTER_OTLP_* variables.
	exporter, err := otlptracehttp.New(ctx)
	if err != nil {
		return fmt.Errorf("create OTLP exporter: %w", err)
	}
	res, err := resource.New(ctx, resource.WithFromEnv(), resource.WithAttributes(
		attribute.String("service.version", "docs-example"),
	))
	if err != nil {
		return fmt.Errorf("create resource: %w", err)
	}
	provider := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter), sdktrace.WithResource(res))
	defer func() {
		flush, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := provider.Shutdown(flush); err != nil {
			log.Printf("flush traces: %v", err)
		}
	}()
	otel.SetTracerProvider(provider)
	otel.SetTextMapPropagator(propagation.TraceContext{})
	tracer := otel.Tracer("avuru-docs/go-example")
	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
		parent := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
		_, span := tracer.Start(parent, "GET /hello", trace.WithSpanKind(trace.SpanKindServer),
			trace.WithAttributes(attribute.String("http.request.method", "GET"), attribute.String("http.route", "/hello")))
		defer span.End()
		span.SetAttributes(attribute.Int("http.response.status_code", http.StatusOK))
		// Kubernetes stdout collection can correlate this line without a second log exporter.
		line, err := json.Marshal(map[string]string{
			"level": "INFO", "message": "hello request handled",
			"trace_id": span.SpanContext().TraceID().String(),
			"span_id":  span.SpanContext().SpanID().String(),
		})
		if err != nil {
			log.Printf("encode log: %v", err)
		} else {
			fmt.Println(string(line))
		}
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		_, _ = fmt.Fprintln(w, "hello from avuru-go-example")
	})
	server := &http.Server{Addr: "127.0.0.1:8081", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	serverErrors := make(chan error, 1)
	go func() { serverErrors <- server.ListenAndServe() }()
	log.Print("listening on http://127.0.0.1:8081/hello")
	select {
	case err := <-serverErrors:
		if !errors.Is(err, http.ErrServerClosed) {
			return fmt.Errorf("serve: %w", err)
		}
	case <-ctx.Done():
		shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		if err := server.Shutdown(shutdown); err != nil {
			return fmt.Errorf("shutdown HTTP server: %w", err)
		}
	}
	return nil
}

func main() {
	if err := run(); err != nil {
		log.Print(err)
		os.Exit(1)
	}
}

The handler extracts W3C traceparent context before starting its server span. The batch exporter sends asynchronously; Ctrl+C gracefully stops the HTTP server and flushes pending spans with a bounded timeout. Only tracing is configured here: metrics need their own meter provider and exporter.

Verify in Avuru Obs

1. Open Traces, choose the receiving project and the last 15 minutes.
2. Filter service avuru-go-example and open GET /hello.
3. Check that the span is a server span and carries http.route=/hello.
4. Compare its trace ID with the JSON line printed by the process.

A laptop's stdout is not automatically sent to a Kubernetes collector. The IDs demonstrate correlation; collecting that line requires a log pipeline. In a pod with stdout collection enabled, the node agent can read it. Avoid also exporting the same log over OTLP unless you intend to ingest it twice. See logs and trace correlation.

If a span does not arrive

• Check exporter errors on stderr and leave the process running long enough for a batch, or stop it gracefully.
• For this HTTP exporter, the signal-specific URL includes /v1/traces; port 4317 is for gRPC.
• Check the ingest key, receiving project, gateway health and current time window.
• HTTPS is supported through the endpoint scheme. Do not disable certificate verification to resolve a configuration error.
• eBPF and SDK spans can overlap. Check which instrumentation observes the same operation before enabling both everywhere.

Continue with distributed tracing, PostgreSQL visibility and the OpenTelemetry Go instrumentation documentation.