Zero-code tracing with eBPF (Linux, beta)

Trace services you cannot change or redeploy. OpenTelemetry eBPF Instrumentation (OBI, Apache-2.0, an OpenTelemetry project) watches running processes from the Linux kernel and sends their requests and database calls to Logthread as OpenTelemetry spans and metrics. You do not touch the code, add a library or restart anything but OBI itself.

This guide installs OBI v0.15.2, the version Logthread has tested, in one of five ways: docker run, Docker Compose, a Kubernetes DaemonSet, a plain Linux host with systemd, or the Logthread Agent's autotrace profile. In the app, Sources → Add source → eBPF (or /onboarding?source=ebpf) shows the same setups with your endpoint filled in and can create the ingest key for you.

Beta. It works as described here on the kernels and runtimes we tested, but it has had less real-world use than the OpenTelemetry SDKs. Tell us what you see on your machines.

What it captures, and what it doesn't#

Captured with no code changes, for any language (Go, Java, Node.js, Python, Ruby, .NET, Rust, C and C++):

• Requests in and out: HTTP/1.1, HTTP/2, gRPC and HTTPS (read before encryption, nothing is decrypted). Server spans fill Endpoints, grouped by route.
• Database and cache calls: PostgreSQL, MySQL, SQL Server and Redis (both sides), plus MongoDB, Elasticsearch and OpenSearch clients. These fill Database.
• Messaging: Kafka, NATS, MQTT and AMQP.
• Metrics: request rate, errors and duration (http.server.request.duration, http.client.request.duration, db.client.operation.duration) in Metrics.

Not captured, or only partly:

• No logs and no custom spans. Use your logging library or a Logthread SDK for logs.
• Code inside a request (functions, loops, in-process caches) is invisible. Only calls that cross a socket are seen.
• Routes are guessed. OBI groups URLs by its own rules (/orders/*), not your framework's routes.
• Some traces are single spans. OBI links services into one trace by adding traceparent to outgoing requests. That always works for Go and Node.js callers over plain HTTP. For Python, Java, Ruby and others it also needs a recent kernel, plain HTTP/1.1 or gRPC (not HTTPS) and no proxy that rewrites requests. Calls between processes on the same host are linked anyway. OBI turns cross-host linking off by itself on kernels where it is unsafe, and logs why. When a link is missing, each service's requests still appear, as separate traces.
• No Windows, no IIS, no macOS. On Windows use the Logthread Agent for logs and metrics and the .NET SDK (Logthread.OpenTelemetry) for traces.

When you need complete traces everywhere, or spans for your own code, use the OpenTelemetry SDK for your language instead. Both can run at once; leave such a service out of OBI's ports so its requests are not traced twice.

Requirements#

• Linux on amd64 (x86_64) or arm64 (aarch64), kernel 5.8 or later with BTF, or RHEL, Rocky or Alma 8+ (kernel 4.18 with the eBPF backports). Ubuntu 20.10+, Debian 11+, Amazon Linux 2023 and current Fedora qualify.
• Root, or the capabilities listed in Kubernetes, and the host's process namespace (pid: host or hostPID: true), because OBI watches other processes. Managed container platforms that forbid this (AWS Fargate, Google Cloud Run, Azure Container Apps) cannot run it.

Check a host:

uname -m -r
ls -l /sys/kernel/btf/vmlinux

A kernel version of 5.8 or later (or 4.18 on RHEL 8) and an existing /sys/kernel/btf/vmlinux mean OBI can run.

The settings every install uses#

Create the ingest key under Keys in the app (it is shown once and is write-only). The examples read it from LOGTHREAD_INGEST_KEY:

export LOGTHREAD_INGEST_KEY=<INGEST_KEY>

Every install also uses this obi.yaml. It picks the processes, skips the container runtime, and turns on two attributes OBI leaves out by default: the statement text for the Database page and peer names for the Service map:

discovery:
  instrument:
    - open_ports: 80,443,8000-8999   # the ports your services listen on
      containers_only: true
  exclude_instrument:
    - exe_path: "{*/dockerd,*/containerd,*/containerd-shim*,*/docker-proxy,*/kubelet,*/logthread-agent*}"
attributes:
  select:
    traces:
      include: [db.query.text, service.peer.name]

• open_ports selects every process listening on one of those ports, including other people's containers on the same host. List your own services' ports.
• containers_only: true keeps OBI to containers. On a plain host, remove that line.
• db.query.text is the statement. Logthread normalises it before display, but the stored attribute can contain literal values from non-parameterised SQL, and Redis keys and values. Remove it from include if that matters; the Database page then has no statements to show.

Service names#

OBI names a service from the process's own environment, in this order:

1. OTEL_SERVICE_NAME (or service.name in OTEL_RESOURCE_ATTRIBUTES) set on that process;
2. in Kubernetes, the pod annotation resource.opentelemetry.io/service.name, then the labels app.kubernetes.io/name and app.kubernetes.io/part-of, then the Deployment, StatefulSet or DaemonSet name;
3. otherwise the executable's name: node, python3.13, java.

Without a name, two Node.js services are both called node and merge into one. Set OTEL_SERVICE_NAME on each service (in its Compose environment:, its systemd unit or its Deployment) and restart it. OBI itself needs no restart: the new process is picked up with the new name.

Docker#

One OBI container per Docker host. Save obi.yaml (above) in the current directory, then:

docker run -d --name obi --pid host --privileged --restart unless-stopped \
  -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/kernel/tracing:/sys/kernel/tracing \
  -v "$PWD/obi.yaml:/etc/obi/obi.yaml:ro" \
  -e OTEL_EBPF_CONFIG_PATH=/etc/obi/obi.yaml \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.logthread.com \
  -e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
  -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $LOGTHREAD_INGEST_KEY" \
  -e OTEL_EBPF_BPF_CONTEXT_PROPAGATION=all \
  docker.io/otel/ebpf-instrument:v0.15.2

Check that it found your services:

docker logs obi 2>&1 | grep "instrumenting process"

Docker Compose#

Add the obi service to your compose.yaml, next to your services (the key comes from LOGTHREAD_INGEST_KEY in the shell or an .env file):

services:
  obi:
    image: docker.io/otel/ebpf-instrument:v0.15.2
    pid: host
    privileged: true
    restart: unless-stopped
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup
      - /sys/kernel/tracing:/sys/kernel/tracing
    environment:
      OTEL_EBPF_CONFIG_PATH: /etc/obi/obi.yaml
      OTEL_EXPORTER_OTLP_ENDPOINT: https://ingest.logthread.com
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${LOGTHREAD_INGEST_KEY}"
      OTEL_EBPF_BPF_CONTEXT_PROPAGATION: all
    configs:
      - source: obi
        target: /etc/obi/obi.yaml

configs:
  obi:
    content: |
      discovery:
        instrument:
          - open_ports: 80,443,8000-8999
            containers_only: true
        exclude_instrument:
          - exe_path: "{*/dockerd,*/containerd,*/containerd-shim*,*/docker-proxy,*/kubelet,*/logthread-agent*}"
      attributes:
        select:
          traces:
            include: [db.query.text, service.peer.name]

docker compose up -d obi
docker compose logs obi | grep "instrumenting process"

Kubernetes#

One OBI pod per node, as a DaemonSet. Save this as obi.yaml and apply it. It uses specific capabilities instead of privileged, as in OBI's own unprivileged example:

apiVersion: v1
kind: Namespace
metadata: { name: logthread }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: obi, namespace: logthread }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: obi }
rules:
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods", "services", "nodes"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: obi }
subjects: [{ kind: ServiceAccount, name: obi, namespace: logthread }]
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: obi }
---
apiVersion: v1
kind: ConfigMap
metadata: { name: obi-config, namespace: logthread }
data:
  obi.yaml: |
    discovery:
      instrument:
        - open_ports: 80,443,8000-8999
          containers_only: true
      exclude_instrument:
        - exe_path: "{*/dockerd,*/containerd,*/containerd-shim*,*/docker-proxy,*/kubelet,*/logthread-agent*}"
    attributes:
      select:
        traces:
          include: [db.query.text, service.peer.name]
---
apiVersion: apps/v1
kind: DaemonSet
metadata: { name: obi, namespace: logthread }
spec:
  selector: { matchLabels: { app: obi } }
  template:
    metadata: { labels: { app: obi } }
    spec:
      serviceAccountName: obi
      hostPID: true
      containers:
        - name: obi
          image: docker.io/otel/ebpf-instrument:v0.15.2
          env:
            - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: https://ingest.logthread.com }
            - { name: OTEL_EXPORTER_OTLP_PROTOCOL, value: http/protobuf }
            - name: LOGTHREAD_INGEST_KEY
              valueFrom: { secretKeyRef: { name: logthread, key: ingest-key } }
            - { name: OTEL_EXPORTER_OTLP_HEADERS, value: "Authorization=Bearer $(LOGTHREAD_INGEST_KEY)" }
            - { name: OTEL_EBPF_KUBE_METADATA_ENABLE, value: "autodetect" }
            - { name: OTEL_EBPF_CONFIG_PATH, value: /etc/obi/obi.yaml }
          securityContext:
            runAsUser: 0
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
              # SYS_ADMIN: trace context across Go services. Add SYS_RESOURCE on kernels before 5.11.
              add: [BPF, PERFMON, NET_RAW, SYS_PTRACE, DAC_READ_SEARCH, CHECKPOINT_RESTORE, SYS_ADMIN]
          volumeMounts:
            - { name: config, mountPath: /etc/obi }
            - { name: var-run-obi, mountPath: /var/run/obi }
            - { name: cgroup, mountPath: /sys/fs/cgroup }
            - { name: tracefs, mountPath: /sys/kernel/tracing }
      volumes:
        - { name: config, configMap: { name: obi-config } }
        - { name: var-run-obi, emptyDir: {} }
        - { name: cgroup, hostPath: { path: /sys/fs/cgroup } }
        - { name: tracefs, hostPath: { path: /sys/kernel/tracing, type: Directory } }

kubectl apply -f obi.yaml
kubectl -n logthread create secret generic logthread --from-literal=ingest-key="$LOGTHREAD_INGEST_KEY"
kubectl -n logthread rollout status ds/obi
kubectl -n logthread logs ds/obi | grep "instrumenting process"

The pod waits for the Secret, so the order of the first two commands does not matter. Spans carry k8s.namespace.name, k8s.pod.name, k8s.deployment.name and k8s.node.name.

Privileged variant. If your cluster's policy allows privileged pods and you would rather not maintain a capability list (or kernel.perf_event_paranoid is 3 or more, as on Debian), replace the whole securityContext: block with:

          securityContext:
            privileged: true

Linux host (systemd)#

For services that run directly on a server. Fetch OBI from its GitHub release and check its checksum:

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSLO https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v0.15.2/obi-v0.15.2-linux-$ARCH.tar.gz
curl -fsSLO https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v0.15.2/SHA256SUMS
grep " obi-v0.15.2-linux-$ARCH.tar.gz\$" SHA256SUMS | sha256sum -c -
sudo tar -xzf obi-v0.15.2-linux-$ARCH.tar.gz -C /usr/local/bin obi

Write the configuration (no containers_only on a host) and the key file, readable only by root:

sudo mkdir -p /etc/obi
sudo tee /etc/obi/obi.yaml >/dev/null <<'EOF'
discovery:
  instrument:
    - open_ports: 80,443,8000-8999
  exclude_instrument:
    - exe_path: "{*/dockerd,*/containerd,*/containerd-shim*,*/docker-proxy,*/kubelet,*/logthread-agent*}"
attributes:
  select:
    traces:
      include: [db.query.text, service.peer.name]
EOF
sudo install -m 600 /dev/null /etc/obi/obi.env
sudo tee /etc/obi/obi.env >/dev/null <<EOF
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.logthread.com
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $LOGTHREAD_INGEST_KEY"
OTEL_EBPF_BPF_CONTEXT_PROPAGATION=all
OTEL_EBPF_CONFIG_PATH=/etc/obi/obi.yaml
EOF

Run it as a service:

sudo tee /etc/systemd/system/obi.service >/dev/null <<'EOF'
[Unit]
Description=OpenTelemetry eBPF Instrumentation for Logthread
After=network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=/etc/obi/obi.env
ExecStart=/usr/local/bin/obi
Restart=on-failure
RestartSec=10s
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now obi
journalctl -u obi | grep "instrumenting process"

Linux host with the Logthread Agent (autotrace)#

If the server runs (or will run) the Logthread Agent, add the autotrace profile instead of installing OBI by hand. The installer downloads OBI as a checksum-verified add-on and runs it as its own root service, logthread-agent-autotrace, next to the agent. OBI sends to the agent on 127.0.0.1:4319, and the agent forwards with its queue and your key, so OBI never holds the key:

curl -sSL https://downloads.logthread.com/agent/install.sh | sudo sh -s -- \
  --key "$LOGTHREAD_INGEST_KEY" --profiles linux-host,autotrace

The ports to trace are LOGTHREAD_AUTOTRACE_PORTS in /etc/logthread-agent/autotrace.env (default 80,443,3000,5000,8000-8999). After changing it:

sudo systemctl restart logthread-agent-autotrace
journalctl -u logthread-agent-autotrace | grep "instrumenting process"

sudo logthread-agent-profile disable autotrace stops it, and enable autotrace starts it again.

Check that it works#

1. OBI found your services. Each install's log shows one instrumenting process line per process it attached to, with the executable and, when known, the service name. No line means no process listens on one of the chosen ports.
2. Send a few requests to your service.
3. About a minute later, open Logthread.

What appears where:

• Traces lists the requests, newest first. A trace from a Go or Node.js caller shows the downstream services and database calls as children.
• Traces → Endpoints shows one row per service, method and route (GET /orders/*), with requests per second, latency and errors.
• Traces → Database shows the statements, normalised (SELECT price FROM items WHERE id = $?), per database system and database.
• Service map draws each service, with an eBPF badge, and its calls to other services, databases and external hosts.
• Metrics has http.server.request.duration and the other OBI metrics.

Every OBI span carries the resource attribute telemetry.distro.name = opentelemetry-ebpf-instrumentation, so you can filter on it in Traces or SQL.

Troubleshooting#

Uninstall#

• Docker: docker rm -f obi
• Docker Compose: remove the obi service, then docker compose up -d --remove-orphans
• Kubernetes: kubectl delete -f obi.yaml
• Linux host: sudo systemctl disable --now obi, then sudo rm -rf /etc/obi /etc/systemd/system/obi.service /usr/local/bin/obi and sudo systemctl daemon-reload
• Logthread Agent: sudo logthread-agent-profile disable autotrace stops it; install.sh --uninstall removes the agent and the add-on

Revoke the ingest key under Keys if no other service uses it.

Next: read the traces in Metrics and traces.