Correlate logs and traces

When a log line carries the trace id and span id of the request that wrote it, Logthread links the two in both directions: from a failing trace to its log lines, and from an error log to the whole trace, across every service the request touched. This page shows how to get the ids onto your logs, how to follow a request, and how to check that it works.

What you get#

Step 1: make the ids reach your logs#

A log record joins a trace when it carries trace_id (32 lowercase hexadecimal characters) and span_id (16). Pick the way that matches how you send logs.

Standard OpenTelemetry, any language#

When you export logs with the OpenTelemetry logs SDK, the SDK stamps each record with the trace context of the active span. Use the same endpoint and key as for traces: see Traces and metrics with OpenTelemetry. Nothing else to do.

The Logthread SDKs#

They attach the active span's ids to each record when the OpenTelemetry API is present, and add a per-request correlation_id. See the Node.js, Python and .NET pages for installing them. Add the request middleware once, per language:

Node.js#

If @opentelemetry/api is installed, the active span's ids are attached to each record. With @logthread/node/otel the same call that sends your traces and metrics starts the spans, so the two meet without more setup:

import { startTelemetry } from '@logthread/node/otel';
import { init, correlation } from '@logthread/node';
startTelemetry({ serviceName: 'checkout' });   // traces and metrics (see the OpenTelemetry page)
const lt = init();                              // logs: a log inside a span carries its ids
app.use(correlation());                         // Express / Connect; Fastify and Koa have their own

Python#

If opentelemetry is installed and a span is active, the record carries its ids. With the logthread[otel] extra the same call that sends your traces and metrics starts the spans:

import logthread
from logthread.otel import start_telemetry

start_telemetry(service_name="checkout")                # traces and metrics (see the OpenTelemetry page)
logthread.install()                                      # logs: a log inside a span carries its ids
app.add_middleware(logthread.CorrelationMiddleware)      # Starlette, FastAPI

.NET#

When a System.Diagnostics.Activity is current, which ASP.NET Core and OpenTelemetry both create, each record gets its traceId and spanId.

builder.Logging.AddLogthread();
builder.Services.AddLogthreadCorrelationForAllHttpClients();   // optional: send the id on outgoing calls
app.UseLogthreadCorrelation();                                  // before routing

JSON or NDJSON#

Write the two keys yourself, and Logthread reads them:

curl https://ingest.logthread.com/v1/ndjson \
  -H "Authorization: Bearer $LOGTHREAD_INGEST_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary $'{"service":"checkout","level":"error","message":"payment declined","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}\n'

If your logs are JSON text in the message, Settings, Data, Parse logs automatically reads trace_id and span_id from it when the record has none. A pipeline's Parse function can do the same for other formats; see Parse logs at ingest.

No trace id available?#

Logs from the same request can still be joined with a request or correlation id. The SDKs add correlation_id; you can also name your own field under Settings, Correlation. Rules apply to newly ingested data only.

Step 2: follow one request#

1. In Explore, search for the failure, for example service:checkout severity:error.
2. Click Show full story. Each card is one request, with its services in order and its duration. First error jumps to the first failing line.
3. When the request has a trace id, open the trace from the card, or switch the timeline to Graph: spans become bars on their service's lane, log events become diamonds, and arrows show each call from a parent span to a child span in another service, in red when the child failed.
4. In the trace, Logs for this trace lists the same lines next to the spans.

Starting from a trace: open it in Traces, find the red span, read its log lines, and use Open in Explore for the full search.

Step 3: check that it works#

Ask the SQL console what share of your logs carry a trace id, by service. A service near 0% is not stamping the ids.

SELECT ServiceName,
  count() AS logs,
  countIf(TraceId != '') AS with_trace,
  round(100 * with_trace / logs, 1) AS percent_with_trace
FROM obs.logs
WHERE Timestamp >= {from:DateTime64(3)} AND Timestamp < {to:DateTime64(3)}
  AND TimestampTime >= toDateTime({from:DateTime64(3)}) AND TimestampTime <= toDateTime({to:DateTime64(3)})
GROUP BY ServiceName
ORDER BY percent_with_trace ASC
LIMIT 50

To see the error logs of the slowest requests next to their spans, use the join in the SQL console cookbook.

When logs do not link#

Next: be told when it breaks, with a delivery channel, in Alerts.