Metrics and traces

Use Metrics to chart any metric your services send, and Traces to find slow or failing requests and see where the time went. The Traces page has two more views built from the same spans: Database (slowest SQL statements) and Endpoints (requests by route). All of them take standard OpenTelemetry data sent to your Logthread endpoint with an ingest key. See Traces and metrics with OpenTelemetry for setup in each language.

Chart a metric#

1. Open Metrics. With no metric chosen, the page lists the metrics seen in the time range under Pick a metric to chart. Choose one, or use the Metric picker and Search metric names.
2. Set the options in the filter bar (below).
3. Drag across the chart to zoom into a time range, or use Zoom out. Use Refresh to reload.
4. The table under the chart lists each series with Last, Min, Max and Avg. Tick or untick a series to show or hide it, or use Show all and Hide all.

If the page says there are no metrics, send some first: Set up metric ingestion opens the setup page. Prometheus remote write keeps counters, gauges and native histograms.

To keep a chart, choose Add to dashboard. To be told when it crosses a line, choose Create alert: it opens a metric threshold rule with the same metric, aggregation, label filters, services and first group-by label, and a starting threshold just above the highest value in the chart (members and above; gauges, counters and histograms).

Filter bar options#

At most 10 series are drawn, taken by most recent value. When more exist, the page says so and suggests a filter or a different grouping. The time range is at most 7 days.

Choose an aggregation (rate and increase)#

The Aggregation list depends on the metric type. Auto picks a sensible default.

For counters, Increase is how much the counter grew in each bucket and Rate (per second) is that growth divided by the time covered. A restart (a new start time or a lower value) is treated as a counter reset, so a restart does not appear as a negative spike. A series that starts inside the window counts its first value. Buckets with no data are zero for rates and increases, and gaps for everything else. Histogram percentiles are interpolated from the bucket counts. Exponential histograms are stored but not charted.

Find a slow trace#

1. Open Traces. The list shows traces newest first, with Start, Root, Duration, Spans, Errors and Services. The duration bar is relative to the slowest trace on the page.
2. Narrow it with the filters:
3. Services: pick up to 20.
4. Operation: the span name, for example GET /checkout.
5. Status: Any, Has errors or OK only.
6. Min (ms) and Max (ms): trace duration bounds.
7. Attribute filter: choose span or resource, enter a key such as http.status_code and a value.
8. Choose Load more for older results. There is no column sort: results are always newest first, so set Min (ms) to isolate slow traces.
9. Select a start time to open the trace.

Three cards above the list show Requests / s, p95 latency and Error rate for the current filters. Export downloads the results. Create alert starts a p95 latency rule for the chosen services and operation, with a threshold 1.5 times the current p95; Add to dashboard adds the p95 chart as a panel (members and above). Status, duration and attribute filters are not carried over.

New traces take about a minute to appear. An empty page offers Set up tracing with a guide per language.

Read a trace#

The trace page shows the trace id (with Copy), Duration, Spans, Errors and Services.

• Waterfall: one row per span, indented by parent. Spans are coloured by service and errors are marked. Click a service in the legend to focus it.
• Drag the overview to zoom, and use Zoom in, Zoom out and Reset zoom. Collapse and Expand fold the span tree. Next error jumps to the next failing span.
• Keyboard: up and down arrows move, left and right collapse and expand, Enter opens details, e jumps to the next error, Esc closes.
• Span details: select a span for its start, duration, span id, parent id, Attributes, Resource, Events and Links. Use Go to parent, or the previous and next arrows.
• Logs for this trace lists logs that carry the trace id, with offset, level, service, message and span. Select a line to highlight its span, or choose Open in Explore. Logs appear when your application writes the trace id into its log records, which the OpenTelemetry SDKs do automatically.
• A truncated trace shows only the first spans it holds, and says so.

If a trace is not found, it may be older than your retention, not ingested yet, or in another workspace. A trace id is 32 lowercase hexadecimal characters.

Find slow database queries#

Open Traces, then the Database tab. It lists SQL statements from database client spans, with parameter values removed (literals become ? and IN lists collapse).

• Filters: Services, Database system, Database, Statement contains, Sort by (Total time, p95, p99, Calls, Errors, Average, Last seen) and Count nested spans. By default only the innermost span of a query is counted, so EF Core plus SqlClient does not double count.
• Columns: Statement, System, Database, Calls, Errors, p50, p95, p99, Total, Last seen.
• Select a row for latency (p50 and p95) and calls per second over time, and the slowest example traces. Add to dashboard adds that statement's p95 and p50 latency as a panel, counted exactly as here.
• What fills it: client spans that carry a statement (db.query.text or db.statement) and a database system (db.system.name or db.system). Database instrumentation in your OpenTelemetry setup (for example pg, psycopg, JDBC, SqlClient, EF Core) provides these.
• From SQL Server shows the most expensive statements reported by the database itself, with executions, elapsed time, CPU and logical reads. It needs the Logthread Agent with the sqlserver profile and top-query collection turned on.

Find slow endpoints#

Open Traces, then the Endpoints tab. It lists server requests by service and route.

• Filters: Services, Method, Route contains, Apdex T (ms) (default 500) and Min requests.
• Columns: Endpoint, Service, Req/s, p50, p95, p99, Errors, Apdex, Slowest (opens the slowest trace). Click a column heading to sort.
• Select a row for requests per second, p50 and p95 latency and error rate over time, plus the slowest example traces. Create alert starts a p95 latency rule on exactly that endpoint (same route matching, HTTP 5xx counted as errors) with a threshold 1.5 times its current p95. Add to dashboard adds its p95 and p50 latency as a panel.
• Apdex counts requests up to T as satisfied and up to 4 times T as tolerating. Bands: 0.94 and up excellent, 0.85 good, 0.70 fair, 0.50 poor, below that unacceptable.
• What fills it: spans of kind server. The route is the http.route attribute, or else the URL path with ids replaced by :id. An error is a span with error status or an HTTP status of 500 or more. HTTP server instrumentation for your framework provides these.

The time range carries over between the Traces, Database and Endpoints tabs.

Next: see what calls what in the service map.