Instrumentation with OpenTelemetry Operator

After installing the OpenTelemetry Operator, you need to configure instrumentation to monitor your applications in specific namespaces. Below, we outline the steps to do this in a simple, efficient way with support for multiple languages.

If you haven’t installed the OpenTelemetry Operator yet, go back to the main guide at: https://docs.elven.works/instrumentacao-em-kubernetes/

Step 1: Create the Instrumentation File

In the stack-observability repository, there is already an instrumentation.yaml file with complete configurations and language-specific examples. Use it as a base and edit it as needed.

Excerpt from the available example:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
  namespace: default
spec:
  exporter:
    endpoint: "http://opentelemetrycollector.monitoring.svc.cluster.local:4318"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"

  go:
    image: "grafana/beyla:latest"
    env:
      - name: BEYLA_METRICS_ENABLED
        value: "true"
      - name: OTEL_METRICS_EXPORTER
        value: "otlp"
      - name: BEYLA_OPEN_PORT
        value: "8080"
      - name: BEYLA_BPF_TRACK_REQUEST_HEADERS
        value: "true"
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://opentelemetrycollector.monitoring.svc.cluster.local:4318"
  nodejs:
    env:
      - name: OTEL_NODE_RESOURCE_DETECTORS
        value: all
      - name: OTEL_TRACES_EXPORTER
        value: "otlp"
      - name: OTEL_METRICS_EXPORTER
        value: "otlp"
      - name: OTEL_NODE_ENABLED_INSTRUMENTATIONS
        value: "all"
      - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
        value: "fs"
      - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: "http://opentelemetrycollector.monitoring.svc.cluster.local:4318/v1/traces"
      - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
        value: "http://opentelemetrycollector.monitoring.svc.cluster.local:4318/v1/metrics"
  java:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://opentelemetrycollector.monitoring.svc.cluster.local:4317
  dotnet:
    env:
      - name: OTEL_DOTNET_AUTO_METRICS_CONSOLE_EXPORTER_ENABLED
        value: "false"

Step 2: Apply the File to the Cluster

kubectl apply -f instrumentation.yaml

Expected result: Your applications will start sending traces and metrics automatically to the Collector.

Step 3: Add Annotations for Automatic Instrumentation

Add specific annotations to Deployment, StatefulSet or Pod to enable automatic instrumentation.

Examples by Language:

annotations:
  instrumentation.opentelemetry.io/inject-nodejs: "true"
  instrumentation.opentelemetry.io/inject-java: "true"
  instrumentation.opentelemetry.io/inject-python: "true"
  instrumentation.opentelemetry.io/inject-dotnet: "true"
  instrumentation.opentelemetry.io/inject-go: "true"

Tip: Add these annotations directly to the application's deployment.yaml file.

Verify the Instrumentation

After adding the annotations, the Operator will automatically inject the agents into the applications.

How to verify:

• Confirm that the pods have sidecars or init containers with the otel agents.

• Check if traces and metrics appear in Grafana or another backend.

Adjusting the Sampling (Sampling Rate)

In the spec.sampler block of the instrumentation.yaml, set the sampling rate to control the amount of traces generated.

sampler:
  type: parentbased_traceidratio
  argument: "1"     # 1 = 100% das requisições, 0.5 = 50%, 0.1 = 10%, etc

Use 100% in test environments and reduce it in production to avoid overhead.

Troubleshooting

Problem	Possible Cause	Solution
Application is not sending traces.	Missing or incorrect annotation.	Check if the Deployment has the correct annotation for the language.
Metrics are not showing up in Grafana.	Missing export environment variables.	Confirm the OTEL_EXPORTER_OTLP_* variables in the language-specific block.
Collector is not receiving data.	Incorrect endpoint or wrong port.	Check the URL http://opentelemetrycollector.monitoring.svc.cluster.local:4318
The agent is not injected.	Wrong namespace in the instrumentation.yaml.	Check if the application's namespace matches the CRD's namespace.
High CPU usage.	100% sampling in production or collection of unnecessary metrics.	Adjust the argument to values like 0.1 and review the enabled exporters.

Reference

• Official Operator Documentation

• Example in the Elven Observability repository

Your application is now ready for true observability with automatic instrumentation and real-time collection of metrics and traces!