Parse logs at ingest

Most logs arrive as one line of text. If you can only search that line, every question turns into a text search. Parsing at ingest turns the line into fields (status, route, duration_ms, ...) while the log is on its way in, so you can filter, group and chart on them right away.

Logthread offers two ways to get fields out of a message:

• At ingest, with an ingest pipeline. The fields are stored with the log. Searches on them are fast, they show up as columns, and alerts and dashboards can use them.
• At query time, with the json, logfmt, regexp and pattern stages of the query language. Nothing is stored extra, and you can change your mind on the next query.

1. Ingest or query time?#

A good habit is to explore with a | regexp or | json stage, and when the query is something you will run again, save it as an ingest pipeline (see section 5).

2. Automatic parsing#

Logthread can parse common log formats for you, with no pipeline: JSON objects, key=value (logfmt) lines, log lines with a timestamp and level in front, and web access logs. Settings, Data, Parse logs automatically turns this on or off for the workspace. New workspaces have it on. Workspaces that existed before the feature keep it off until an admin turns it on. One setting covers all these formats and the severity rules below.

Each log is tried in this order, and the first format that fits is used: JSON, then access log, then a line with a timestamp and level in front, then key=value. A message that fits none is left as it is. The message text itself is never removed (except for JSON with a message field, below), so you can always see the original line.

JSON#

When it is on and the message starts with {:

• every field is stored as an attribute, using dotted names for nested objects (user.id), down to 5 levels. Arrays and deeper levels are stored as JSON text;
• a non-empty message or msg becomes the log message. The original JSON is then not kept, because every field is already an attribute. Without one (or when fields were truncated) the original JSON stays as the message;
• level, severity or lvl sets the severity when the log has none, as a word (warn, error) or a number on the pino scale (10 trace to 60 fatal);
• trace_id and span_id set the trace and span ids when they are empty;
• timestamp, time, ts or @timestamp sets the event time when the log has none, as ISO 8601 or a number (seconds, milliseconds, microseconds or nanoseconds, read from its size). These fields, the level and the ids are also kept as attributes;
• attributes your collector already set win over fields of the same name in the body, and names starting with logthread. are ignored;
• if a key appears twice, the first one wins; messages larger than 64 KiB are left as they are, invalid JSON leaves the log unchanged, and at most 128 fields are added per log (the record is then marked logthread.json_truncated=true).

The fields are stored, so stored volume can grow. If you only need a few fields, leave the setting off and use a pipeline with a parse function plus remove, or filter at query time.

Key=value (logfmt)#

A message such as

level=info msg="hello world" status=200 route=/pay

becomes the attributes level, msg, status (a number) and route, so service:api route=/pay status>=400 works without | logfmt. A message counts as key=value only when it is strict about it:

• it is a single line with at least two key=value pairs, and the pairs make up at least 70% of the text. Keys use letters, digits, _, ., - and / and start with a letter or _. Values may be quoted (msg="hello world", with \" for a quote);
• prose that happens to contain an = is left alone (retrying after error code=5), and so are URLs with query strings (GET /search?q=a&x=1) and lines cut off inside a quoted value;
• numbers and true/false are stored as numbers and booleans, everything else as text;
• the same rules as JSON apply: attributes your collector already set win, names starting with logthread. are ignored, the first of two equal keys wins, messages over 64 KiB are left alone, and at most 128 fields are added (then the log is marked logthread.json_truncated=true).

Lines with a timestamp and level in front#

Most server loggers write a prefix before their key=value pairs. Logthread reads the prefix and then the pairs at the end of the line. The message text stays as it is, and the words between the prefix and the first pair are not stored.

The rules are strict, so ordinary text is not mistaken for a log line:

• the line starts with a date and time (ISO 8601, a T or a space, a comma or a dot before the fraction, an optional zone) followed within a few words by a level: INFO, WARN, WARNING, ERROR, ERR, CRIT, FATAL, DEBUG, TRACE and so on, in any case, also as [INFO] or INFO:. A thread in brackets, a process id, ---, a logger name and - or : separators may follow;
• at least two key=value pairs (or a JSON object) must end the line, with the same key and quoting rules as key=value above. If there are fewer, only the severity (and time) is set. Text with an = and no such prefix is left alone;
• the level in the prefix is what the log said, so the severity is not marked logthread.severity_inferred. A severity your collector already set is never changed;
• the event time is taken from the line only when the log has none and the time states a zone (Z or an offset such as +05:30). A time without a zone, which Python's default format writes, is local time of an unknown machine, so the log keeps the time it arrived;
• for a multi-line message (a stack trace) only the first line is read;
• the same limits as key=value apply: attributes your collector already set win, a logger attribute you set wins over the one in the prefix, names starting with logthread. are ignored, and at most 128 fields are added.

Web access logs#

Lines in the common or combined log format, which includes the nginx and Apache defaults:

203.0.113.7 - frank [10/Oct/2026:13:55:36 +0000] "GET /pay/checkout?id=7 HTTP/1.1" 200 2326 "https://shop.example.com/cart" "Mozilla/5.0"

are stored with OpenTelemetry attribute names:

The status must be 100 to 599 and the time stamp must parse, so other text is not mistaken for an access log. The query string and the user name are not stored. If the log has no time, the time in the line is used.

Severity#

A log with no severity gets one when Logthread can tell what it is. A severity your collector or the log sent is never changed.

• If the log has a level, severity or lvl attribute (for example from key=value or JSON), a known level word sets the severity: trace, debug, info, notice, warn/warning, error/err, critical, alert, emergency, fatal/panic, in any case.
• Otherwise an HTTP status sets it. The status comes from the access log, or from an attribute named http.response.status_code, http.status_code, status or status_code that holds a number from 100 to 599. 5xx is ERROR, 4xx is WARN, everything else is INFO.

A severity set this way is marked with the attribute logthread.severity_inferred=true, so you can tell it from one that was sent. Use severity:error and the severity facet as usual: service:nginx severity:warn now finds the 4xx requests.

The fields are stored, so stored volume can grow. If you only need a few fields, turn the setting off and use a pipeline with a parse function plus remove, or filter at query time.

Automatic parsing runs before your pipelines, so a pipeline can mask or rename the fields it produced.

3. The parse function#

An ingest pipeline is an ordered list of functions (see Pipelines in the app). The parse function reads one field (the message by default) and writes the fields it finds as attributes. A message that does not match is left unchanged; it is never an error and never dropped.

Common settings:

Formats#

• json: the message is a JSON object. Nested keys become dotted names.
• logfmt: key=value key2="quoted value" pairs, as written by Go and many CLI tools.
• kv: key and value separators you choose (kv_pair, kv_sep), for example a:1;b:2.
• regex: an RE2 regular expression with named groups, (?P<status>\d{3}). Each named group becomes an attribute.
• grok: patterns such as %{IP:client} %{WORD:method} %{URIPATH:path}. Logthread supports a built-in set of names (numbers, words, IPs, hostnames, URIs, quoted strings, timestamps, log levels, and the timestamp and level formats of nginx, Apache, Postgres, Kubernetes and Redis). Names it does not know are rejected when you save. Add :int or :float to read a value as a number, %{NUMBER:bytes:int}: values that are not numbers are skipped, and a value of - sets nothing.
• delimited: one value per column, for CSV, TSV, pipe-separated and space-aligned logs.

Regular expressions and grok patterns use RE2, so there is no backtracking: a pattern that is accepted cannot make ingest slow.

Delimited#

A line with fewer values than columns sets only the values present. A value that is just - means "no value" and sets nothing, which is how many access logs write an empty field. An unterminated quote leaves the log unchanged.

Example, a CSV log with a quoted message:

{"type": "parse", "format": "delimited", "separator": ",",
 "columns": ["ts", "level", "user", "message"], "extra": "last"}

4. The parser library#

You do not have to write patterns for common formats. In the app, open Pipelines, Library (or Pipelines, Add from library), search by name or category and look at the sample lines before and after parsing. Use this parser creates a draft pipeline. The draft changes nothing for your incoming logs. Preview it on real logs in the editor, then add it to a route and activate the routes.

Every parser is tested against sample lines. Where a format has a log level, the parser sets the severity. Where it has a time, the parser sets the event time. Numbers such as status codes, byte counts and durations are stored as numbers, so you can compare and aggregate them.

The exact attribute names each parser writes are shown in its sample preview. If your log format differs slightly (a custom NGINX log_format, for instance), use the closest parser as a starting point and edit the pattern in the pipeline editor. Your edits are saved as a new version, and you can roll back at any time.

5. From Explore to a pipeline#

Found a query that parses your logs the way you want? In Explore, open the Pipeline editor and choose Save as ingest pipeline (admins). Logthread converts:

Everything else is listed as not carried over: where filters are not turned into drops, and fields, sort, limit and stats only shape query results. The service and severity terms of your search become a suggested route filter. Other search terms are not carried over.

The converted pipeline is validated and saved as a draft. Nothing changes for incoming logs until you add it to a route and activate the routes. Open the draft in Pipelines to preview it on real logs first.

6. Limits and performance#

• Attribute values are stored as text. Number comparisons still work in searches and queries (| where status >= 500, | stats p95(duration)): Logthread reads the text as a number at query time.

• Ingest pipelines run in the worker, after the log is safely queued. A pipeline can never make a batch fail: a function that cannot read a log leaves it as it was.
• Parsing only affects logs that arrive after a pipeline is active. Logs already stored keep their old shape.
• Ingest is metered before pipelines run. Dropping or sampling reduces what is stored and queried, not the billable ingest.
• A pipeline has at most 50 functions and 64 KB of configuration. A workspace has at most 20 pipelines.
• Automatic JSON and the delimited parser do not use regular expressions and are the cheapest ways to parse. Grok and regex parsers are cheap too, but a very long pattern costs more per log than a short one, so anchor patterns with ^ where you can.
• Measured speed (one core of a development machine, Apple M4; the Ampere servers Logthread runs on are slower per core): automatic JSON takes about 2 µs for a 1 KB message, key=value about 0.6 µs, a line with a prefix about 0.5 µs, an access log about 0.8 µs and a message that fits no format about 45 ns; the delimited parser about 4 µs for 20 columns; the nginx access-log parser about 5 µs. A whole worker pass with automatic JSON, a parser, correlation and patterns handles about 230,000 logs per second per core.
• Attribute names are stable across logs. Avoid parsing values that change per request (ids, timestamps) into attribute names.

Next: find what costs the most and trim it in Cost control.