Logthread collector pack for LiveKit

Ships a self-hosted LiveKit deployment (Server, SIP, Egress, Ingress) to Logthread:

• Metrics: the Prometheus endpoints of each service, scraped by an OpenTelemetry Collector.
• Logs: the services' container logs (zap JSON or console), parsed and mapped to Logthread's LiveKit attribute names (livekit.room.sid, livekit.room.name, livekit.participant.*, ...). The room session id also becomes correlation_id, so one room session reads as one story.
• Events: LiveKit webhooks go straight to Logthread (step 3 below).

Files: otel-collector.yaml (config), compose.yaml (drop-in service), helm-values.yaml (Kubernetes), pipelines/livekit-agent-logs.json (preset for agent logs), METRICS.md.

Set it up in three steps#

1. Change the LiveKit config. Each setting below is verified against the LiveKit docs and a running container.

livekit.yaml (LiveKit server):

prometheus:
  port: 6789          # serves http://<host>:6789/metrics
logging:
  level: info
  json: true          # zap JSON, one object per line

LiveKit SIP (config.yaml or SIP_CONFIG_BODY):

prometheus_port: 6790
health_port: 8081     # optional
log_level: info       # debug | info | warn | error

SIP writes zap *console* lines (<ts>\t<LEVEL>\t<logger>\t<caller>\t<msg>\t{json}); the pack parses both formats, so no change is needed.

Egress and Ingress configs: set prometheus_port (Egress default scrape target :6789, Ingress :7889 in the pack; change the *_TARGET variables if yours differ).

2. Run the collector. Create an ingest key in Logthread (Keys, "Create an ingest key for LiveKit"), then:

export LOGTHREAD_ENDPOINT=https://<your-ingest-host>      # e.g. the gateway base URL, no /v1/logs
export LOGTHREAD_INGEST_KEY=<your ingest key>
# merge compose.yaml into your LiveKit compose project, and put otel-collector.yaml next to it
docker compose up -d logthread-collector

Each LiveKit service needs a label so its logs are picked up and named, and the json-file driver to expose it:

  livekit:
    labels:
      logthread.service: livekit-server    # livekit-sip | livekit-egress | livekit-ingress
    logging:
      driver: json-file
      options:
        labels: logthread.service

Scrape targets come from env (LIVEKIT_SERVER_TARGET, LIVEKIT_SIP_TARGET, LIVEKIT_EGRESS_TARGET, LIVEKIT_INGRESS_TARGET, each host:port). Delete the jobs for components you do not run; they only log scrape warnings. On Kubernetes use helm-values.yaml with the open-telemetry/opentelemetry-collector chart.

3. Point LiveKit webhooks at Logthread. In livekit.yaml:

webhook:
  api_key: <your livekit api key>
  urls:
    - <ingest-base>/v1/integrations/livekit/<connector-id>

<ingest-base> is your gateway base URL and <connector-id> comes from the LiveKit connector page in Logthread (create the connector there with your API key and secret; the secret is stored encrypted). Webhooks give room, participant, track, egress and ingress events, including SIP call attributes on participants.

Stack traces and multi-line logs#

Docker and Kubernetes store a stack trace one line per entry. The pack joins every line that does not start a new record (a JSON object, a timestamp, a level word or panic:) onto the record before it, per container. So a livekit-sip error with its stack, a Go panic or a Python traceback arrives as one log. The message stays the body and the stack goes to exception.stacktrace. A record is sent once the next record starts, or after 1 second of quiet.

Upgrading from 0.1.0: replace otel-collector.yaml (or helm-values.yaml) with this version and restart the collector (docker compose up -d logthread-collector, or helm upgrade). A collector managed by Logthread Fleet gets the new LiveKit template from Fleet → Configs.

Check it works#

• Explore: filter service = livekit-server. A participant joining shows participant active with livekit.room.name, livekit.room.sid, livekit.participant.*, correlation_id.
• Metrics: search livekit_ (see METRICS.md for the names that exist).

LiveKit Cloud#

Cloud has no Prometheus endpoint or container logs for you to collect, so this collector pack does not apply. Use:

• Webhooks: Cloud project settings, add <ingest-base>/v1/integrations/livekit/<connector-id> as an endpoint (the connector uses the project's API key and secret).
• Log drain for cloud-hosted agents: syslog over TCP+TLS to Logthread's syslog endpoint, with your ingest key as SYSLOG_TOKEN (the connector page prints the exact command).
• Agents code: the Logthread LiveKit Agents helpers (logthread-livekit for Python, @logthread/livekit-agents for Node) add traces, usage metrics and logs with one call. Apply pipelines/livekit-agent-logs.json to parse JSON agent logs and drop lk.pii.* fields.
• Analytics (Scale plan): enable "Analytics" on the connector for per-session summary logs.

Agent logs preset#

pipelines/livekit-agent-logs.json is a Logthread pipeline ({"functions": [...]}): parses JSON bodies, uses message/msg as the body, maps room/participant/job fields to livekit.*, sets correlation_id to the room sid, removes lk.pii.* attributes, and masks card numbers and secrets in the body. Create it from Pipelines, "Import JSON", or POST /api/pipelines plus a version, then add a route that matches your agent's service.name. To keep lk.pii.* keys but hide their values, replace the remove function with a mask on attributes.* fields.

Notes and limits#

• Log timestamps come from Docker's capture time, not the zap ts field (milliseconds apart).
• start_at: end: the collector ships lines written after it starts.
• The collector needs read access to /var/lib/docker/containers (root). With the local or journald Docker log drivers, use the journald/syslog receivers instead.
• Never commit the ingest key; pass it by environment variable.