Docker hosts: container logs with the fluentd log driver

Send the logs of containers on a plain Docker or Docker Compose host to Logthread without reading files on the host. The Docker fluentd log driver forwards each container's output to an OpenTelemetry Collector on the same machine, and the collector sends it to Logthread over OTLP/HTTP with an ingest key.

container stdout/stderr → Docker fluentd log driver → collector (127.0.0.1:8006) → https://ingest.logthread.com

Use this when you run docker run or Compose on a VM and cannot (or would rather not) mount /var/lib/docker/containers. For Kubernetes use the Helm chart under Sources → Kubernetes and Docker; to collect host metrics and Windows or Linux logs as well, use the Logthread Agent.

This guide assumes the OpenTelemetry Collector Contrib distribution (otelcol-contrib), version 0.111.0 or newer. The fluentforward receiver, file_storage extension and the other components used here are all in that distribution, not in the core one. Check the Collector documentation for newer versions.

1. Create an ingest key#

Under Keys in the app, create an ingest key. It is shown once and is write-only. Put it in an env file that only root can read:

sudo install -d -m 755 /etc/logthread
echo 'LOGTHREAD_INGEST_KEY=<INGEST_KEY>' | sudo tee /etc/logthread/collector.env >/dev/null
sudo chmod 600 /etc/logthread/collector.env

2. The collector configuration#

Save this as /etc/logthread/otelcol.yaml:

extensions:
  file_storage:
    directory: /var/lib/otelcol/queue   # holds the disk-backed send queue
    create_directory: true

receivers:
  fluentforward:
    endpoint: 127.0.0.1:8006            # loopback only; never expose this port

processors:
  transform/docker:
    log_statements:
      - context: log
        statements:
          # The driver sends {log, container_id, container_name, source}. Keep them as attributes
          # and make the message text the log body.
          - merge_maps(attributes, body, "insert") where IsMap(body)
          - set(body, attributes["log"]) where attributes["log"] != nil
          - delete_key(attributes, "log")
          # container_name starts with "/": strip it to get the service name.
          - set(attributes["service.name"], attributes["container_name"]) where attributes["container_name"] != nil
          - replace_pattern(attributes["service.name"], "^/", "")
  groupbyattrs:
    keys: [service.name]                # one resource per container
  resource:
    attributes:
      - key: host.name
        value: ${env:HOST_NAME}
        action: upsert
  batch:
    timeout: 2s
    send_batch_size: 2048

exporters:
  otlphttp:
    endpoint: https://ingest.logthread.com
    compression: gzip
    headers:
      Authorization: Bearer ${env:LOGTHREAD_INGEST_KEY}
    sending_queue:
      enabled: true
      storage: file_storage             # survives collector restarts and outages
      queue_size: 5000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0               # keep retrying; the disk queue holds the data

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [fluentforward]
      processors: [transform/docker, groupbyattrs, resource, batch]
      exporters: [otlphttp]

• service.name is the container name without the leading /, so a container started with --name checkout is the service checkout. Compose names are <project>-<service>-<n>; to get a stable name, set container_name: in the Compose file, or set tag (below) and adapt the transform.
• host.name comes from the HOST_NAME variable you pass when you start the collector.
• Use the base endpoint only: the exporter adds /v1/logs.

3. Run the collector#

docker run -d --name logthread-collector --restart unless-stopped \
  --network host \
  --log-driver json-file \
  --env-file /etc/logthread/collector.env \
  -e HOST_NAME="$(hostname)" \
  -v /etc/logthread/otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro \
  -v logthread-queue:/var/lib/otelcol \
  --user 0:0 \
  otel/opentelemetry-collector-contrib:0.111.0

• --network host lets the collector listen on the host's 127.0.0.1:8006, which is where the Docker daemon connects. If you prefer, drop it and publish a loopback port instead: -p 127.0.0.1:8006:8006 with endpoint: 0.0.0.0:8006 in the receiver.
• --log-driver json-file on the collector matters if you later set the fluentd driver daemon-wide: the collector must not send its own logs to itself.
• The named volume logthread-queue keeps the disk queue across restarts. --user 0:0 lets a fresh volume be written; you can run as a non-root user if you prepare the volume's permissions.

4. Send a container's logs#

Per container:

docker run -d --name checkout \
  --log-driver fluentd \
  --log-opt fluentd-address=127.0.0.1:8006 \
  --log-opt fluentd-async=true \
  --log-opt tag='{{.Name}}' \
  my-image:latest

Docker Compose:

services:
  checkout:
    image: my-image:latest
    container_name: checkout
    logging:
      driver: fluentd
      options:
        fluentd-address: 127.0.0.1:8006
        fluentd-async: "true"
        tag: "{{.Name}}"

All containers on the host, in /etc/docker/daemon.json:

{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "127.0.0.1:8006",
    "fluentd-async": "true",
    "tag": "{{.Name}}"
  }
}

Then sudo systemctl restart docker. This applies only to containers created afterwards, and restarting the daemon stops containers unless live-restore is on. Start the collector first (it uses json-file, so it comes up either way).

Always set fluentd-async=true. Without it the driver blocks, and a container that cannot reach the collector will not start.

Note that with the fluentd driver docker logs <container> still works on current Docker Engine releases through dual logging; on older ones it does not.

5. Give plain-text lines a severity#

Container output is text, so lines have no level. Add the Severity from text function in Pipelines (also in Pipelines → Library) so ERROR: ..., [warn] ... and level=error ... at the start of a line set the severity. It only fills in a severity the record does not already have.

If your containers print JSON, key=value (logfmt) or access-log lines, automatic parsing already turns the fields into searchable attributes and picks up level, so you do not need a function for those. See Pipelines and parsing and Parse logs at ingest.

6. Check that it works#

1. Open Sources → Add source (or the Docker card under Integrations); the first-data check reports when the first record arrives.
2. In Explore, search service:checkout (use your container name).
3. Open Live tail and filter to the same service to watch lines arrive.

If nothing shows, make the container write a line to its main output (restarting it usually does), then see Troubleshooting.

Troubleshooting#

• The container will not start or hangs: "failed to initialize logging driver". The driver cannot connect to 127.0.0.1:8006. Start the collector first and check docker logs logthread-collector. Set fluentd-async=true so a down collector does not block the container.
• Port 8006 is not listening. With --network host check ss -ltn | grep 8006. Without it, the port must be published on loopback.
• Records arrive with service.name empty or unknown_service. The record had no container_name. Check that the daemon is a version that sends it and that you did not remove it with fluentd-labels or env options.
• Collector restarts and the queue. The sending_queue stores batches on the file_storage volume, so a restart or a Logthread outage does not lose what was queued. If the volume is lost, queued data is lost. Check that /var/lib/otelcol/queue is writable and has space.
• HTTP 429 in the collector log. You went over your plan's ingest rate. The collector retries with backoff and the queue absorbs bursts; see Usage and your plan limits, or reduce volume with a pipeline drop or sample function.
• HTTP 401 or 403. The key is wrong, revoked or has trailing whitespace in the env file. Create a new key under Keys, update /etc/logthread/collector.env and run docker restart logthread-collector. Never put the key in the YAML file or in a Compose file you commit.
• Everything shows up as one blob of text. Multi-line output (stack traces) arrives as one line per line printed unless the application writes one record per event. Log JSON, or join lines in the application.

Next: find your container logs in Explore and live tail.