Back to the blog Infrastructure

Observability with OpenTelemetry: where to start

What traces, metrics and logs are, what the Collector is for and how to instrument, sample and name services with OpenTelemetry. Includes a Collector configuration, a Python example, a staged plan and a checklist.

David López · CEO - Fundador November 4, 2026 10 min read

A customer complains that checkout is slow. The application server has logs, the database has metrics and the load balancer has yet another console. Nothing links a slow request to the query that held it up, and each tool uses its own agents and formats. Switching vendors means instrumenting everything again.

OpenTelemetry (OTel) solves the collection side: it defines an API, an SDK per language, a protocol (OTLP) and an intermediate service, the Collector, to generate traces, metrics and logs and send them to any compatible backend. The CNCF describes it as a vendor-neutral observability framework and moved it to Graduated in May 2026.

This guide explains each signal, the Collector, instrumentation and cost control, and proposes a staged plan. The versions cited are those current as of November 4, 2026.

Three signals and how they connect

OpenTelemetry defines signals as system outputs that describe its activity. The three main ones are:

Traces. The path of a request through your application. A trace is made of spans, each one a unit of work, with a shared trace ID and its own span ID. A span has attributes (key-value pairs) and a status: Unset by default, Error or Ok.
Metrics. Measurements captured at runtime: counters, gauges and histograms, among others.
Logs. The record of an event. OpenTelemetry does not ask you to change your logging library: it connects to it through bridges.

Correlation is what makes the set useful. When a span is active, the SDK adds the TraceId and SpanId fields to each log record, so you can jump from a log to the trace of that request. In metrics, exemplars tie an aggregated data point to the trace ID and span ID of a specific measurement. Across services, context travels in headers: by default, SDKs use the tracecontext (W3C Trace Context) and baggage propagators.

To start measuring, Google's SRE book proposes four golden signals for user-facing systems: latency, traffic, errors and saturation. It recommends distinguishing the latency of successful requests from that of failed ones.

The Collector: a layer between your apps and the backend

The Collector receives, processes and exports telemetry without tying you to a vendor. It takes work off the application: retries, batching, encryption and sensitive data filtering. The documentation recommends running a Collector alongside your services; sending directly to the backend is fine to get started or for development.

In its configuration, receivers take in data, processors modify it in the order you list them and exporters send it to one or more destinations. Under service.pipelines you define one pipeline per signal. A component that is configured but does not appear under service is not enabled.

This example fits a central Collector in a small company. It uses the otelcol-contrib distribution, because the transform processor is not part of the core distribution.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:            # always first
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  transform/sensitive:       # drops an attribute that must not leave the app
    error_mode: ignore
    trace_statements:
      - delete_key(span.attributes, "customer.tax_id")
    log_statements:
      - delete_key(log.attributes, "customer.tax_id")

exporters:
  otlp_grpc/traces:          # e.g., Jaeger
    endpoint: jaeger:4317
    tls:
      insecure: true         # only inside a private network
    sending_queue:
      batch: {}
  otlp_http/metrics:         # e.g., Prometheus with its OTLP receiver
    endpoint: http://prometheus:9090/api/v1/otlp
    sending_queue:
      batch: {}
  otlp_http/logs:
    endpoint: http://logs-backend:4318
    sending_queue:
      batch: {}

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/sensitive]
      exporters: [otlp_grpc/traces]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_http/metrics]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/sensitive]
      exporters: [otlp_http/logs]

Five details of the current version, 0.164:

Since 0.144, the otlp exporter is called otlp_grpc and otlphttp became otlp_http. The old names remain as deprecated aliases; it is worth migrating.
Batches are built in the exporter itself with sending_queue and batch: {}. The project has already released the queue_batch processor, which replaces the legacy batch processor; it is in beta and not yet included in the official distributions.
The memory_limiter goes first in every pipeline, and its documentation also recommends setting the GOMEMLIMIT variable on every Collector.
The Collector listens on localhost by default. 0.0.0.0 exposes it to the whole network; use it only if other machines must send it data, and protect that access.
Validate the file with otelcol-contrib validate --config=config.yaml.

There are two basic ways to deploy it. As an agent, on the same host as the application: easy to start, but it scales poorly. As a gateway, a central point: it centralizes credentials and filtering or sampling policies, at the cost of another component to maintain, some added latency and more resources.

Automatic and manual instrumentation

Zero-code instrumentation captures telemetry from the libraries and frameworks you already use, without touching your code. Code-based instrumentation adds what only your application knows, such as which order failed and why. You can use both at the same time.

Before choosing, check the status of each signal in your language:

Language	Traces	Metrics	Logs
Java	Stable	Stable	Stable
C#/.NET	Stable	Stable	Stable
PHP	Stable	Stable	Stable
Go	Stable	Stable	Release candidate
JavaScript	Stable	Stable	Development
Python	Stable	Stable	Development
Ruby	Stable	Development	Development

The same page warns that, whatever that status, instrumentation that relies on experimental semantic conventions may change in breaking ways.

A Python example. The current stable SDK is 1.45.2 (it requires Python 3.10 or later); the agent and instrumentation libraries are still published as beta, at 0.66b2. opentelemetry-bootstrap reads the installed packages and installs the matching instrumentation, for example the one for Flask.

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_SERVICE_NAME=store-api
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.8.3,deployment.environment.name=production
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
opentelemetry-instrument python app.py

With that you already get spans for HTTP requests and for the instrumented libraries. To add a business operation, use the API:

from opentelemetry import metrics, trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("store.orders")
meter = metrics.get_meter("store.orders")
orders = meter.create_counter(
    "store.orders", unit="{order}", description="Orders processed"
)

def process_order(order):
    # If the block raises an exception, the span records it and is set to Error.
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("store.order.channel", order.channel)
        result = charge(order)
        if not result.approved:
            span.set_status(Status(StatusCode.ERROR, "payment declined"))
        orders.add(1, {"store.order.channel": order.channel,
                       "store.payment.approved": result.approved})

In Python, start_as_current_span records the exception and marks the span as an error by default; there is no need to repeat it. Python logs are still in Development: log auto-instrumentation is enabled by default since 1.40.0, and OTEL_PYTHON_LOG_CORRELATION=true injects trace context into log records.

Sampling, cardinality and costs

By default, the SDK uses parentbased_always_on: it keeps every trace. The documentation suggests not sampling if you generate little data, on the order of tens of small traces per second or less, and considering it from about 1,000 traces per second.

Head sampling. The decision is made when the trace starts. It is simple and efficient, but it cannot guarantee that every trace with an error is kept. With OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.25 you keep one in four root traces and respect the calling service's decision.
Tail sampling. The decision is made with the complete trace, in the Collector. It lets you keep all errors and slow requests, at the cost of memory and complexity.

processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: rest
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

The tail_sampling processor keeps spans in memory while it decides: by default it waits 30 seconds (decision_wait) and holds up to 50,000 traces (num_traces). All spans of a trace must reach the same instance. With more than one Collector you need a front layer with the load-balancing exporter and routing_key: traceID, as the gateway pattern describes.

In metrics, cost is driven by cardinality: the number of distinct attribute combinations. A user ID or a raw URL path as an attribute can cause unbounded memory growth. The SDK caps each metric stream at 2,000 combinations and groups the rest under otel.metric.overflow=true; from then on, any query that filters by that attribute undercounts. Use attributes with few values, like the channel in the example, and use Views to trim what you do not need.

Finally, minimize data: collect only what serves an observability purpose and avoid personal data. Legal compliance is the implementer's responsibility, not the tool's.

Semantic conventions and resource attributes

Semantic conventions set common names for operations and data. The conventions for HTTP spans are stable and use http.request.method, url.path, http.route and http.response.status_code. The span name should be {method} {route} only if the route is low-cardinality, for example GET /orders/{id}; never the raw URL path.

The resource describes the entity producing telemetry. The key attribute is service.name: it is required, it must be the same across all replicas and, if you do not set it, the SDK uses unknown_service plus the executable name. Add service.version and deployment.environment.name to separate production from testing. Prometheus, for example, builds its job and instance labels from service.name, service.namespace and service.instance.id.

Open source backends

Because OTLP is a standard, changing backends means changing a Collector exporter, not re-instrumenting. Some open source options that accept OTLP natively:

Traces: Jaeger, whose version 2 is a distribution of the Collector itself and stores data in Cassandra, Elasticsearch, OpenSearch or ClickHouse, among others.
Metrics: Prometheus, with the OTLP receiver enabled through --web.enable-otlp-receiver.
Logs and more: the project's list marks which solutions are open source and which accept OTLP natively. It is not exhaustive and no longer accepts new entries.

Choose based on what your team can operate: each backend is another service to maintain.

A staged plan for a small company

Names first. Define service.name, service.version and deployment.environment.name for each service.
One Collector and debug. Deploy a Collector and instrument one or two critical services with zero-code instrumentation. The debug exporter shows what arrives before you choose a backend.
Traces and metrics. Connect a traces backend and a metrics backend. Build dashboards and alerts on the four golden signals.
Correlated logs. Send logs with TraceId and SpanId. For logs already written to files, the Collector has a file receiver.
Business spans. Instrument by hand only where automatic instrumentation falls short.
Volume control. Review cardinality and decide whether you need sampling.

If you run Kubernetes, the observability section of Kubernetes in production: 9 mistakes complements this plan.

Checklist

Area	Check
Resource	Explicit `service.name` on every service; none shows up as `unknown_service`
Environment	`deployment.environment.name` separates production from testing
Collector	`memory_limiter` first, `GOMEMLIMIT` set and configuration validated
Network	Ports 4317 and 4318 exposed only to those who must send data
Names	Exporters without deprecated aliases (`otlp` → `otlp_grpc`, `otlphttp` → `otlp_http`); the `otlp` receiver keeps its name
Signals	Status of traces, metrics and logs reviewed for each language
Correlation	Logs with `TraceId` and `SpanId`; W3C propagation between services
Cardinality	No user IDs or raw URL paths as metric attributes
Sampling	Written decision: no sampling, head or tail, based on volume
Sensitive data	Attributes with personal data removed in the Collector
Alerts	Latency, traffic, errors and saturation for each critical service

Sources

If you want to work out where to start with your team, let's talk.