TL;DR — Quick Fix
Get distributed tracing running in under 10 minutes with OpenTelemetry auto-instrumentation:
# Node.js — zero-code instrumentation
npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
# Set environment variables and run
export OTEL_SERVICE_NAME=my-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
node --require @opentelemetry/auto-instrumentations-node/register app.js
# Python — auto-instrument with one command
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
export OTEL_SERVICE_NAME=payment-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
opentelemetry-instrument python app.py
---
Architecture — How Traces Flow Across Services
---
Step 1 — Deploy the OpenTelemetry Collector
The Collector acts as a central pipeline for receiving, processing, and exporting telemetry data.
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
tail_sampling:
decision_wait: 10s
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow-traces
type: latency
latency: {threshold_ms: 1000}
- name: probabilistic-sample
type: probabilistic
probabilistic: {sampling_percentage: 10}
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
# docker-compose.yaml — OTel stack
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.96.0
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "8889:8889" # Prometheus metrics
depends_on:
- tempo
- loki
tempo:
image: grafana/tempo:2.4.0
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
ports:
- "3200:3200"
grafana:
image: grafana/grafana:10.3.0
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
---
Step 2 — Instrument Your Services
Node.js (TypeScript)
// tracing.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'api-gateway',
[ATTR_SERVICE_VERSION]: process.env.APP_VERSION || '1.0.0',
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317',
}),
instrumentations: [getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-fs': { enabled: false },
})],
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown());
Go Service
// tracing.go
package tracing
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)
func InitTracer(ctx context.Context, serviceName string) (*sdktrace.TracerProvider, error) {
exporter, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint("otel-collector:4317"),
otlptracegrpc.WithInsecure(),
)
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String(serviceName),
)),
sdktrace.WithSampler(sdktrace.ParentBased(
sdktrace.TraceIDRatioBased(0.1),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}
---
Step 3 — Correlate Trace IDs with Logs
Inject the trace ID into every log line so you can jump from logs to traces:
# Python — structured logging with trace context
import logging
from opentelemetry import trace
class TraceIdFilter(logging.Filter):
def filter(self, record):
span = trace.get_current_span()
ctx = span.get_span_context()
record.trace_id = format(ctx.trace_id, '032x') if ctx.trace_id else '0'
record.span_id = format(ctx.span_id, '016x') if ctx.span_id else '0'
return True
logger = logging.getLogger(__name__)
logger.addFilter(TraceIdFilter())
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
'{"time":"%(asctime)s","level":"%(levelname)s","msg":"%(message)s",'
'"trace_id":"%(trace_id)s","span_id":"%(span_id)s"}'
))
logger.addHandler(handler)
---
Step 4 — Sampling Strategies for Production
| Strategy | Use Case | Overhead |
|---|---|---|
| Head-based (ratio) | General traffic | Low |
| Tail-based (collector) | Keep errors + slow traces | Medium |
| Parent-based | Respect upstream decisions | Low |
| Always-on | Debug environments only | High |
# Tail sampling — keep all errors + 10% of everything else
processors:
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
- name: keep-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-slow
type: latency
latency: {threshold_ms: 2000}
- name: sample-rest
type: probabilistic
probabilistic: {sampling_percentage: 10}
---
Step 5 — Deploy Grafana Tempo for Trace Storage
# tempo.yaml
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.amazonaws.com
region: us-east-1
wal:
path: /tmp/tempo/wal
block:
bloom_filter_false_positive: 0.05
querier:
max_concurrent_queries: 20
metrics_generator:
storage:
path: /tmp/tempo/generator/wal
traces_storage:
path: /tmp/tempo/generator/traces
---
Frequently Asked Questions
How much performance overhead does OpenTelemetry add?
With head-based sampling at 10%, overhead is typically under 2% CPU and 50MB additional memory per service. The biggest cost is context propagation in high-throughput services (100K+ req/s), where you should use the batch span processor with conservative batch sizes.
Should I use Jaeger or Grafana Tempo?
Tempo is preferred for new deployments because it uses object storage (S3/GCS) instead of Elasticsearch/Cassandra, reducing operational overhead by 80%. Jaeger is better if you already run Elasticsearch. Both support OTLP natively.
How do I trace across message queues (Kafka, RabbitMQ)?
Inject trace context into message headers. OpenTelemetry has instrumentation libraries for Kafka (opentelemetry-instrumentation-kafka-node) and RabbitMQ that automatically propagate traceparent headers through message producers and consumers.
What's the difference between head-based and tail-based sampling?
Head-based sampling decides at trace start (cheap but random). Tail-based sampling waits until the trace completes, then decides based on attributes like error status or duration. Tail-based is more useful but requires buffering traces in the collector.
How do I connect traces to metrics in Grafana?
Enable exemplars in Prometheus. When a trace is sampled, the OTel SDK attaches the trace ID as an exemplar to related metrics. In Grafana, click an exemplar point on a metrics graph to jump directly to the corresponding trace.
---