Alerts and incidents

An alert rule watches your logs, metrics or traces and opens an incident when something looks wrong. Logthread then notifies the delivery channels you chose, and closes the incident when things recover. Open Alerts to manage Rules, Incidents and Channels. Every incident links to its full story in Explore.

Choose a rule type#

A rule's type cannot change after it is created. Create a new rule instead.

Set up an alert#

1. Open Alerts, then Rules, then New alert rule (Create a rule when there are none yet). You need the member role or higher; viewers can read rules and incidents.
2. Walk through the five steps: Type, Conditions, Preview, Notify, Name.
3. On Preview, read the backtest ("Would this have fired?") and choose Looks right. If it fires too often, loosen the thresholds. A count rule replays every evaluation of the last 24 hours exactly. A computed value says whether it crosses the limit now and how often it would have fired in the last 24 hours: a percentage replays every evaluation, while another aggregate (p95, average and so on) is checked over back-to-back windows, so the rule itself can fire a little more often. An error spike looks at the last 3 hours of per-minute counts.
4. On Notify, tick channels, pick a Severity (Info, Warning, Critical) and Repeat while still firing.
5. Give the rule a Rule name, leave Enabled ticked and choose Create rule.

If no channel exists, the rule still opens incidents but nobody is told. Add a channel first (see below).

You can also start a rule from the page you are looking at, with its conditions filled in:

• Explore: Create alert (scheduled search on the current query).
• Patterns: Alert on a pattern row.
• Metrics: Create alert (metric threshold on the chart's metric, aggregation, filters, services and first group-by).
• Traces: Create alert (p95 latency for the chosen services and operation).
• Traces → Endpoints: Create alert in a selected endpoint (p95 latency of exactly that route).

Conditions for each rule type#

New pattern: Services (empty means all) and Minimum severity (Any severity, Warning and above, Error and above, Fatal only). A new workspace is given an hour to learn what is normal first, so it does not alert on every line.

Error spike: Services, Window (minutes) (1 to 60, default 5), Baseline (minutes) (5 to 170, default 60), Factor (default 3) and Minimum errors (default 20). It fires when errors in the window are at least the factor times the usual rate and at least the minimum count, and resolves after three calm checks.

Scheduled search: choose Match records by:

• A search (the default): the same search you type in Explore, with fields, numeric comparisons, and, or and a leading minus, optionally followed by parse and where stages (not stats). For example service:gateway gw.status>=500 counts gateway responses with a 5xx status, from a parsed or OpenTelemetry attribute. Try it in Explore opens the search there. The search is checked when you save, with the same messages as Explore.
• Simple fields: Message contains (case-insensitive text), Services, Severities (none selected means any) and the field filters carried over from an Explore search.

Then Fire when the match count is greater than, at least, less than or at most a Count, Over the last (min) (5 to 1440) and Check every (min) (1 to 60). Tick Evaluate each service separately for one incident per service.

Metric threshold: pick a Metric, an Aggregation valid for its type, Services, up to 10 Label filters, and optionally Evaluate separately for each service or label value. Then set the operator, Threshold, Over the last (min) (1 to 60), Check every (min) (1 to 60) and Minimum data points.

• Aggregations: gauge (Average, Minimum, Maximum, Sum, Last value), counter (Rate per second, Increase, plus the gauge set for non-monotonic sums), histogram (p50, p90, p95, p99, Average, Rate).
• The value is computed the same way as on the Metrics page for the same window, so the alert value matches the chart. A group with no data in the window is never a breach. A group with fewer points than Minimum data points is ignored.
• Exponential histograms and summaries are not supported by alerts.

Trace latency or errors: Services, Operation contains, a Measure (p95 latency, p99 latency, Median latency, Average latency, Error rate, Error count, Request rate), optional per Service or Operation grouping, then the operator, Threshold (milliseconds, a percentage for error rate, a count, or per second), Over the last (min) and Check every (min), each 1 to 60.

Scheduled searches, metric thresholds, trace rules, absence rules and computed-value rules resolve after one clean check.

Absence alerts#

An absence rule tells you when a source stopped sending. Pick A service stopped sending on the Type step, then set:

• Watch by Service or by Host (the host.name resource attribute).
• Names: the services or hosts to watch (up to 50), or All names seen to watch everything that sent data in the lookback.
• Quiet for (min): 5 to 1440. A name that has sent nothing for this long fires.
• Lookback (h): 1 to 168. Only names seen at least once in this window can fire; a name that has never sent data, or last sent longer ago than the lookback, never opens an incident.
• Check every (min): 1 to 60.

Logthread takes the newest timestamp of any log, span or metric point for each name, so a service counts as alive if it sends any of the three. Each quiet name opens its own incident: "api has sent nothing for 17 minutes (last seen 2026-10-27 09:43 UTC)". The incident links to Explore for that name around the time it was last seen, and resolves on its own after the name sends data again.

Example: to be told if api, worker or gateway sends nothing for 15 minutes, choose Service, enter api, worker and gateway as names, and set Quiet for to 15 and Lookback to 24.

Alert on a computed value or a percentage#

A scheduled search only counts records. Pick A computed value or a percentage on the Type step to alert on a number worked out from your logs, such as "p95 of gw.duration_ms above 5000" or "more than 2% of requests are server errors". It uses the same query language as Explore, so the value the rule checks is the value you see there for the same query and window. The editor offers two starting templates, p95 latency above N and Error rate above N%.

Choose Alert on:

• A computed value: write a query that ends in exactly one aggregate: count, count_distinct, sum, avg, min, max, p50, p90, p95 or p99. Leave out the by clause; tick Evaluate each service separately to get one value, and one incident, per service.
• A percentage: give the Records that count (a selector, with optional parse and where stages, and no stats) and All records. The value is 100 × matching ÷ all, in percent, so the threshold is 0 to 100. When there are no records in the window the rate has no value and nothing fires.

Then set the operator, the threshold, Over the last (min) (1 to 1440), Check every (min) (1 to 60) and, optionally, Only with at least (records). With a minimum, a window with fewer matching records (the query before its aggregate, or All records for a percentage) has no value, so one failed request out of two never fires an error-rate rule while traffic is low. The rule fires while the value satisfies the condition and resolves after one clean check. Preview shows the current value, or one per service, before you save.

Examples, for a gateway that logs gw.status and gw.duration_ms:

• p95 latency above 5 seconds: query service:gateway | stats p95(attr.gw.duration_ms), greater than 5000, over the last 10 minutes. The incident reads "p95(attr.gw.duration_ms) is 6230 (> 5000) over 10 minutes".
• Server error rate above 2%: Alert on a percentage, records that count service:gateway gw.status>=500, all records service:gateway, greater than 2, over the last 10 minutes, only with at least 50 records. The incident reads "Error rate is 3.4% (> 2%) over 10 minutes".
• More than 20 server errors in 10 minutes: a Scheduled search with the search service:gateway gw.status>=500, at least 20, over the last 10 minutes.

Every incident links to Explore with the query over the window. Percentiles are exact up to 8,192 matching values per group in a window and sampled above that, so very large windows can differ from an exact figure by a small amount. A record whose field is not a number is left out of the aggregate. The query must be valid LTQL: a bad query, two aggregates, a by clause or a percentage threshold above 100 is rejected with a message that names the field.

Choose severity and repeats#

• Severity is a label on the incident: Info (worth knowing), Warning (needs a look soon), Critical (needs action now; pages on-call when routed to PagerDuty).
• Repeat while still firing sends a reminder every 15 minutes, 30 minutes, 1, 2, 4, 12 or 24 hours (default 1 hour). Acknowledging an incident stops the reminders.
• A resolve message is sent when an incident clears, except for new-pattern rules.

Add a delivery channel#

Under Alerts, Channels, choose Add your first channel or Add a channel. Creating, editing, deleting and testing channels needs the admin role.

• Secrets are stored encrypted and shown only as their last characters. Leave a secret blank when editing to keep it.
• Private and internal addresses are refused for webhook URLs.
• Send test queues a test message. The Delivery log shows each notification's status (Pending, Sent, Failed), attempts and last error. A failed delivery is retried with a growing delay, up to 8 attempts.
• Deleting a channel removes it from every rule that uses it.

Move alerts from Grafana#

On Alerts, choose ⋮ then Import from Grafana and paste or upload an alert rules export (Grafana: Alerting → Alert rules → Export, JSON or YAML).

• A PromQL rule (query, reduce, threshold) becomes a metric threshold rule: the metric name, its type and its labels are matched to what you send, and the threshold, window and evaluation interval carry over.
• A LogQL count_over_time or rate rule becomes a scheduled search; an unwrapped quantile or average becomes a computed value rule.
• labels.severity sets the severity, and a paused rule comes over disabled.
• Math expressions, classic conditions with several parts and range thresholds are listed as not supported, with the reason.

Each rule is previewed on your data first ("would have fired 3 times recently"). Contact points become channels: email keeps its addresses. Grafana never exports secrets such as Slack webhook URLs or PagerDuty keys, so paste them in the import, or leave one blank and add it later in Channels. Rules send to the channels made from their contact point.

Work with incidents#

Open Alerts, then Incidents. Filter by Firing, Resolved or All, and by rule. Each row shows its scope, Started, Duration and Value against the threshold.

• A firing incident can be Acknowledged (member role). The row then shows "Acked by", and repeat notifications stop. There is no manual resolve: an incident resolves itself when its rule stops firing.
• A rule that fires again within 10 minutes of resolving re-opens the same incident.
• View story opens the matching window in Explore.
• Open an incident for its state, severity, value versus threshold, Evidence (a chart around the firing window), Sample events captured when it opened, and the Notifications sent.
• If the rule was muted when the incident opened, it is marked "Firing · muted" and no notifications are sent.

New-pattern incidents resolve on their own after 24 hours, without a resolve notification.

See what changed#

An incident's page lists the deploys, config changes and flags from the hour before it, scored against the service that broke. Every change is under Alerts, Changes. See What changed before an incident.

Mute, disable or delete a rule#

On Rules, each row has Disable / Enable and an actions menu with Edit, Mute for 1 hour, Mute for 8 hours, Mute for 24 hours (or Unmute) and Delete. A muted rule still evaluates and records incidents but sends no notifications. Deleting a rule removes its incidents and notification history.

Limits: 200 rules per workspace, of which 50 can be scheduled (scheduled search, metric threshold or trace rules), and 50 channels. Each rule can use up to 20 channels.

Create alerts from Explore, CI or an AI agent#

• From Explore: Create alert opens a new rule with the search exactly as it is in the search bar. A search that ends in one aggregate (| stats p95(...)) opens a computed-value rule; any other search opens a scheduled search. Preview runs it against your data before you save, and warns when it matched nothing in the last 24 hours, naming any field that no record carries and similar ones that do.
• From CI or a script: create an API key with Create alert rules (the alerts scope) and POST /api/alerts/rules:

curl -X POST https://logthread.com/api/alerts/rules \
  -H "Authorization: Bearer $LOGTHREAD_API_KEY" -H 'Content-Type: application/json' \
  -d '{"name": "Gateway 5xx", "type": "scheduled_query", "severity": "critical",
       "config": {"filter": {"query": "service:gateway gw.status>=500"},
                  "window_min": 10, "every_min": 5, "condition": {"op": ">=", "value": 20}}}'

A computed value or percentage is "type": "query_value" with {"query", "ratio_of"?, "op", "threshold", "window_min", "every_min", "by_service"?, "min_count"?}. POST /api/alerts/rules/preview with {"type", "config"} returns the preview, including matched_24h (for a ratio rule, the denominator's count) and, for a ratio rule, ratio_24h: the numerator, the denominator, their ratio and the highest ratio of any window in the last 24 hours, with its time and service. The rule editor's preview shows the same line. The key can create rules, not change or delete them.

• From an AI agent: the MCP tool create_alert_rule takes the same name, type, config, severity and channels, with a key that has the alerts scope. See AI agents (MCP).

Next: send incidents to your own systems with alert webhooks, then chart the numbers on a dashboard.

SLOs and error budgets#

An SLO is an objective for the share of requests that succeed, such as 99.5% of checkout requests over 30 days. Open Alerts → SLOs, choose New SLO and give:

• All requests (total): an LTQL selector, for example service:checkout. Parse and where stages are allowed; stats is not.
• Failed requests (bad): the failing ones among them, usually the same selector plus a condition, for example service:checkout status>=500.
• Target: the percent that must not fail, from 90 to 99.999.
• Window: 7, 28 or 30 days, rolling.

Check on your data shows the numbers before you save. Logthread counts both selectors exactly, the same way alert rules count, and shows:

• the SLI (good ÷ total);
• the error budget left (the failures the target allows, minus those that happened);
• the burn rate over the last hour (1× spends the budget exactly over the window);
• a daily burn-down.

It warns when the total matches nothing, or when bad is larger than total on some day. It also says what the bad records are, for example "67 of 76 bad records in the 30-day window have code=operation_not_supported". Count only real failures: a scanner asking for operations you do not offer, or a client cancelling, is not your service failing, so leave it out of bad (service:gateway status>=500 code!=operation_not_supported).

Add burn alerts creates two ordinary alert rules on the share of failed requests, so they preview, mute and notify like any rule:

Both rules need at least 50 requests in their window (you can change it when adding them), so one failure among a few requests on a quiet night never pages anyone. Changing an SLO carries the change to its burn rules (selectors, threshold, name); they keep their channels, minimum and mutes. Deleting an SLO can delete its burn rules too. An SLO without burn alerts shows "Not alerting" until you add them. A workspace can have 50 SLOs. The API is GET/POST /api/slos, GET/PUT/DELETE /api/slos/{id}, POST /api/slos/preview and POST /api/slos/{id}/alerts; AI agents use list_slos, create_slo and delete_slo.