Kubernetes: your apps, traces and correlated logs

There are two separate jobs in a cluster, and each has its own page:

• The cluster itself: pod logs, node and pod metrics, restarts and deployment health. Install the Kubernetes integration (two Helm releases of the OpenTelemetry Collector chart) from Integrations. See Kubernetes in the Integrations guide.
• Your applications: traces, metrics and logs that carry the same trace id, sent straight from the app. This page.

You can do either one, or both.

1. Keep the ingest key in a Secret#

Create an ingest key under Keys in the app (it is shown once and can only send), then store it in your cluster. Do not put it in an image or a manifest.

kubectl create secret generic logthread --from-literal=ingest-key="$LOGTHREAD_INGEST_KEY"

2. Send OTLP from the app#

Standard OpenTelemetry settings do all the work. Add them to the Deployment of each service. The Downward API fills in the pod, namespace and node, so every signal says where it ran.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 2
  selector:
    matchLabels: {app: checkout}
  template:
    metadata:
      labels: {app: checkout}
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.4.0
          env:
            - name: LOGTHREAD_INGEST_KEY
              valueFrom:
                secretKeyRef: {name: logthread, key: ingest-key}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
            - name: POD_NAMESPACE
              valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
            - name: NODE_NAME
              valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
            - name: OTEL_SERVICE_NAME
              value: checkout
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: https://ingest.logthread.com
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: http/protobuf
            - name: OTEL_EXPORTER_OTLP_HEADERS
              value: "Authorization=Bearer%20$(LOGTHREAD_INGEST_KEY)"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),k8s.node.name=$(NODE_NAME)"
            - name: OTEL_LOGS_EXPORTER
              value: otlp

Kubernetes expands $(NAME) from variables defined earlier in the same list, so keep LOGTHREAD_INGEST_KEY, POD_NAME, POD_NAMESPACE and NODE_NAME above the settings that use them. Bearer%20 is a URL-encoded space, as the OpenTelemetry SDKs expect. The language-specific part (the agent or SDK your app starts with) is in Traces and metrics with OpenTelemetry. Give each service its own OTEL_SERVICE_NAME.

Outbound HTTPS on port 443 to ingest.logthread.com is all your cluster needs. Nothing is opened inbound.

3. Check that data arrives#

1. Open Traces and look for your service. Spans appear within a minute or so.
2. In Explore, search a resource attribute: res.k8s.namespace.name:default service:checkout.
3. In Metrics, pick a metric from your service.

If nothing arrives, follow the checklist in Send data: a wrong key answers 401 and a blocked egress shows as a timeout in the app log.

4. Logs that link to traces#

A log line joins its trace when it carries the trace id and span id. In Kubernetes you can get them there in two ways. Pick one per service.

Send logs over OTLP (recommended). With OTEL_LOGS_EXPORTER=otlp, as above, and a logging library bridged to OpenTelemetry, each record carries the trace context of the active span. These logs go from the app to Logthread directly, with the same resource attributes as your traces. The settings are the same ones, and the full explanation is in Correlate logs and traces.

Write JSON to stdout and let the cluster integration collect it. Make your logger include trace_id and span_id in each JSON line. With Parse logs automatically on (Settings, Data), Logthread reads them from the message. The node collector from the Kubernetes integration then ships the line with its pod and namespace.

Do not do both for the same service: the same line would be stored twice, and stored lines are not deduplicated.

5. Name things so you can find them#

• Set OTEL_SERVICE_NAME on every Deployment, so each service has its own name in Logthread and two services never merge under a default name.
• Keep the cluster name consistent. The Kubernetes integration asks for an optional cluster name and shows it as k8s.cluster.name. Add it to OTEL_RESOURCE_ATTRIBUTES on your apps too.
• For services you cannot change, zero-code tracing runs as a DaemonSet and needs no code changes.

Next: follow a failing request across services in Explore and live tail.