TIL: using the debug exporter in OTEL collector to diagnose the filter processor

We have a bunch of microservices that use OpenTelemetry for tracing. We forward all our traces to a self-hosted OTEL collector, which in turn forwards everything to ElasticSearch.

I’ve been trying on and off again to filter our http requests we don’t care about, but this proved to be a bit of a pain because I wasn’t entirely sure what attributes I should filter on.

For this you can use a debug exporter, which shows the traces being handled in the console:

processors:
  batch:
  memory_limiter:
    limit_mib: 1500
    spike_limit_mib: 512
    check_interval: 5s
  filter:
    error_mode: ignore
    traces:
      span:
        - IsMatch(attributes["http.route"], "^.*\\/actuator.*$")
        - IsMatch(attributes["http.route"], "^.*\\/bulk.*$")
        - IsMatch(attributes["http.route"], "^.*\\/api\\/events\\/raw.*$")
        - IsMatch(attributes["http.route"], "api/checkup")
        - IsMatch(attributes["url.full"], ":9200/_bulk$")

exporters:
  otlp/elastic:
    ...
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

service:
  extensions: [zpages, memory_ballast]
  pipelines:
    traces/1:
      receivers: [otlp]
      processors: [memory_limiter, batch, filter]
      exporters: [debug, otlp/elastic]

The output looks like this:

Span #10
    Trace ID       : 6b1913cc1cd5c6e440050d4beb4e3dbb
    Parent ID      :
    ID             : d583049c8c9dc8a7
    Name           : <redacted>
    Kind           : Client
    Start time     : 2024-05-28 09:49:53.8791287 +0000 UTC
    End time       : 2024-05-28 09:49:53.8800347 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> db.system: Str(mssql)
     -> db.name: Str(<redacted>)
     -> net.peer.ip: Str(<redacted>)
     -> db.statement_type: Str(Text)
     -> db.statement: Str(<redacted>)
     -> peer.service: Str(<redacted>)