How do you implement distributed tracing and observability in a Go microservice system?

Observability in distributed systems requires three pillars: metrics (what is happening?), logs (what happened?), and traces (why is a specific request slow?). OpenTelemetry is the standard SDK for Go.

// OpenTelemetry setup
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (func(), error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil { return nil, err }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("user-service"),
            semconv.ServiceVersionKey.String("1.0.0"),
        )),
        trace.WithSampler(trace.TraceIDRatioBased(0.1)), // sample 10%
    )
    otel.SetTracerProvider(tp)
    return func() { tp.Shutdown(context.Background()) }, nil
}

// Instrument a function
var tracer = otel.Tracer("user-service")

func (s *userServiceServer) GetUser(
    ctx context.Context, req *pb.GetUserRequest,
) (*pb.User, error) {
    ctx, span := tracer.Start(ctx, "UserService.GetUser")
    defer span.End()

    span.SetAttributes(
        attribute.Int64("user.id", req.Id),
    )

    user, err := s.repo.FindByID(ctx, int(req.Id))
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return nil, status.Errorf(codes.Internal, "internal error")
    }
    return toProto(user), nil
}

// Propagate trace context in gRPC metadata
// Use otelgrpc interceptors to do this automatically:
grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

Correlation IDs: every request entering the system gets a trace ID propagated through gRPC metadata, HTTP headers, and message queue headers. This enables you to see the full call tree of a single request across 10 services in a tool like Jaeger or Tempo.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.