Complete Guide to Instrumentation and Monitoring in Kubernetes

This guide was created to help you easily set up a monitoring and instrumentation stack using modern tools such as Prometheus, Promtail, and OpenTelemetry. You will use Grafana from Elven Observability to visualize your metrics, logs, and dashboards at the following address:

https://grafana.elvenobservability.com

Repository Content

stack-observability/
├── opentelemetry-operator/
│   ├── instrumentation.yaml
│   ├── values.yaml
│   ├── README.md
├── otel-collector/
│   ├── collector-config.yaml
│   ├── collector-deploy.yaml
│   ├── collector-service.yaml
│   ├── kustomization.yaml
│   ├── secrets.env
│   ├── README.md
├── prometheus/
│   ├── values-prometheus.yaml
├── promtail/
│   ├── values-promtail.yaml
├── helmfile.yaml
└── README.md

Prerequisites

Make sure you have the following configured in your environment:

• Kubernetes: Functional and configured cluster.

• kubectl: To manage resources in Kubernetes.

• Helm: To install charts .

• Helmfile: To install multiple charts with ease.

Install Helmfile:

curl -sL <https://github.com/helmfile/helmfile/releases/download/v1.0.0-rc.7/helmfile_1.0.0-rc.7_linux_amd64.tar.gz> | sudo tar -xz -C /usr/local/bin

Installation: Step by Step

Clone the Repository

Via SSH

git clone [email protected]:elven-observability/stack-observability-k8s.git

Via HTTPS

git clone <https://github.com/elven-observability/stack-observability-k8s.git>

Configure Namespace and Credentials

Create the monitoring namespace

kubectl create ns monitoring

Next, configure a Secret to store the Elven Observability credentials. These credentials may have been provided directly by our team, or automatically generated if you registered through the website: http://elvenobservability.com/. If you haven’t received them yet, please contact support.

kubectl create secret generic elven-observability-credentials \
  -n monitoring \
  --from-literal=tenantId="<YOUR_TENANT_ID>" \
  --from-literal=apiToken="<YOUR_API_TOKEN>"

Important: Replace <YOUR_TENANT_ID> and <YOUR_API_TOKEN> with the correct values received during onboarding or via registration.

Configure Prometheus

Edit the file prometheus/values-prometheus.yaml:

remoteWrite:
  - url: <https://mimir.elvenobservability.com/api/v1/push>
    authorization:
      type: Bearer
      credentials:
        key: apiToken
        name: elven-observability-credentials
    headers:
      X-Scope-OrgID: <SEU_TENANT_ID>
    relabelConfigs:
      - sourceLabels: [__name__]
        regex: "^(prometheus|go|promhttp|scrape).*"
        action: drop

Promtail Configuration: Filters by Namespace or Annotation

In the promtail/values-promtail.yaml, edit:

config:
  snippets:
    common:
      # Filtro por annotation
      # - action: keep
      #   source_labels: [__meta_kubernetes_pod_annotation_promtail_logs]
      #   regex: "true"

      # Filtro por namespace
      - action: keep
        source_labels: [__meta_kubernetes_namespace]
        regex: "^(default|monitoring|namespace1|namespace2)$"

How to Use:

• By Namespace: collects all logs from pods in the defined namespaces.

• By Annotation: collects only from pods with the annotation promtail_logs: "true":

metadata:
  annotations:
    promtail_logs: "true"

Configure the OpenTelemetry Operator

Edit opentelemetry-operator/instrumentation.yaml:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
  namespace: default
spec:
  nodejs:
    env:
      - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
        value: fs
      - name: OTEL_NODE_RESOURCE_DETECTORS
        value: all
  exporter:
    endpoint: "<http://opentelemetrycollector.monitoring.svc.cluster.local:4318>"
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"

Apply to the cluster:

kubectl apply -f opentelemetry-operator/instrumentation.yaml

Refer to the README.md file in the opentelemetry-operator folder pfor more instrumentation examples.

Install the Components with Helmfile

helmfile sync

Access to Grafana

Access: https://grafana.elvenobservability.com

Use the credentials provided via support or after registration on the website to view dashboards for metrics, logs, and traces.

Installed Resources

• Prometheus: Collection of metrics and alerts

• Promtail: Collection of logs

• OpenTelemetry Operator: Automatic instrumentation

• OpenTelemetry Collector: Centralization of metrics and traces

Examples and Best Practices

• Ready-made examples in the opentelemetry-operator/ folder

• Whenever possible, use annotation instead of capturing logs from the entire namespace

• Adjust trace sampling according to the criticality of the application

• Organize dashboards in Grafana by application or business domain

Tips and Optimizations

• Use relabelConfigs to avoid excessive data ingestion

• Use custom attributes (resource_attributes) to enrich trace and metric data

• Explore public dashboards on Grafana for inspiration

• Add alerts based on critical metrics (e.g., 5xx errors, high latency, absence of events)

Troubleshooting (Problem Solving)

Problem	Common Cause	Solution
Data does not appear in Grafana	Misconfigured secret or invalid token	Check the Secret confirm the credentials.
Logs are not reaching Loki.	Filters in Promtail or missing annotation	Review the filtering rules and annotations.
Missing traces	Incorrect collector endpoint.	Check the URL in theinstrumentation.yaml

Official Documentation

• OpenTelemetry

• Prometheus

• Loki

• Grafana

Support

If you have any questions, difficulties, or suggestions:

• Open an issue in this repository.

• Contribute with a Pull Request.

• Talk to the Elven team.