Pipelines: parse, mask, drop and sample logs

Ingest pipelines change your logs on the way in, before they are stored: parse a message into fields, mask sensitive data such as card numbers and emails, drop noise like health checks, or keep only a sample. A pipeline is an ordered list of functions. Routes decide which records go through which pipeline. This page covers what pipelines can do and how to set one up. For the parsing formats and patterns, see Parse logs at ingest.

Where to find pipelines#

Open Pipelines in the app. It has four tabs:

Who can use them#

What a pipeline can do#

Add functions with Add function. They run in the order shown on every record routed to the pipeline.

Every function has an optional Filter (a CEL expression such as service == "nginx" && severity_number >= 17) so it only touches matching records, a Description, and an enable checkbox. Use the arrows to reorder them.

Mask sensitive data#

The Mask function applies to the fields you list (for example body, or attributes.* for every attribute value). Tick the detectors you want:

• card, aadhaar, pan, email, phone_in, ipv4, jwt, bearer, aws_key, secret_kv, signed_url.
• Or add up to 10 Custom patterns (RE2 regular expressions).

Choose a Mode: redact replaces the match with [REDACTED:name], partial keeps the last 4 characters, and hash replaces it with a stable per-workspace hash so values can still be matched across logs.

signed_url masks the values of these query parameters in URLs and query strings, and leaves the parameter names: X-Amz-Signature, X-Amz-Credential, X-Amz-Security-Token, sig, signature, token and access_token. Names match in any letter case. For example GET /file?X-Amz-Signature=3f9c&X-Amz-Expires=60 becomes GET /file?X-Amz-Signature=[REDACTED:signed_url]&X-Amz-Expires=60.

Every record a Mask function changes carries the attribute logthread.masked, a comma-separated list of the detectors that fired (for example bearer,email). If several Mask functions change the same record, the lists are merged. Records that were not changed do not get the attribute. Find redacted records with attr.logthread.masked, and open any record in Explore to see which detectors fired.

Redact secrets in every route#

The Pipelines page has a Redact secrets in every route switch (admins and owners). Turn on masks bearer tokens, JWTs, AWS keys, secret key=value pairs, card numbers, email addresses and signed-URL signatures in the message and every attribute, for every record:

• it runs after whichever route handled the record, so fields that route parsed are masked too;
• it also covers records that no route matches, and routes you add later;
• the card says what it covers ("2 routes and records no route matches") and which version runs, with Edit detectors and, after you save a new version, Update to vN.

The first time, it creates the Redact secrets pipeline from the Library. Turn off stops masking new records and keeps the pipeline. It applies to records that arrive from now on; stored logs are not changed. A record it changed shows Redacted: email, bearer… in its details.

Create a pipeline#

1. Open Pipelines and click New pipeline.
2. Enter a Name and an optional Description, then Create. It starts empty, which passes records through unchanged.
3. Use Add function to build the list.
4. Try it in Sample and preview: Paste lines with one log line per row, or load recent logs From Explore with a query and a range. The panel shows each record before and after, with counts per function.
5. Click Save new version and add an optional note. Every save is a new version; earlier ones stay in Version history, and admins can restore one.

Or start from the Library: find your format, check the before and after sample, and click Use this parser to create a draft pipeline.

A saved pipeline does nothing until it is active. Its state shows in the list and at the top of the editor: Active (with its routes), Not active: no route uses it → Activate, or v2 running, v3 is saved → Update. Unfinished pipelines are also listed in Finish setting up on Home.

A pipeline has at most 50 functions and 64 KB of configuration. A workspace can have at most 20 pipelines.

Turn it on#

Saving a pipeline changes nothing for incoming logs until it is active. Activate (in the editor, the list or Finish setting up) walks you through it in one dialog:

1. Where: Every record (after any route, and records no route matches; the default for a pipeline of Mask functions), Records from chosen services (pick from the services that sent data in the last hour, with their counts), or Instead of the pipeline on existing routes.
2. Check the match: how many records from the last hour would go through it, with a warning when none would.
3. Preview: the fields each recent record would gain or lose.
4. Review and activate: the exact route changes, applied at once. Routes that ran an older version move to the latest.

Routes by hand#

1. Open Routes and click Add route.
2. Enter a Name, an optional filter (CEL, empty matches every record) and the pipeline with its pinned version.
3. Order the routes: a record uses the first route it matches. Records that match no route are stored unchanged (the Passthrough row at the end). A route set to Every record (the Runs column) instead runs on every record its filter matches, after the first-match route; redaction uses this.
4. Click Review and activate, check the changes and click Activate.

A route keeps running the version it pins. After saving a new pipeline version, pin it in the route and activate again. Activation applies to new records within seconds. The route history lets you roll back to an earlier set. A workspace can have at most 20 routes, and a pipeline that active routes use cannot be deleted.

Turn an Explore query into a pipeline#

If a query in Explore already parses your logs the way you want, an admin can save it as an ingest pipeline from the pipeline editor. It becomes a draft you preview and route as above. The conversion rules are in Parse logs at ingest.

Panels and alerts from parsed fields#

In Sample and preview, every attribute the pipeline added or changed has a Use menu in the After column. A field whose sample values are all numbers is a number; anything else is a string. The menu opens the dashboard panel editor or the alert rule editor already filled in. Nothing is saved until you save there. The query names the attribute exactly as stored, so a field parsed with the prefix gw. is attr.gw.status.

• Add to dashboard, number: | stats p50(attr.gw.ms), p95(attr.gw.ms), max(attr.gw.ms) by bin(ts, 5m).
• Add to dashboard, string: | stats count() by bin(ts, 5m), attr.gw.account.
• Group by: the menu lists the other parsed string fields; the panel then adds that field to the by list (a number shows its p95 per value).
• Create alert: the menu lists what fits the field. A field named status, *.status or *status_code offers a count of 5xx records (more than 10 in 10 minutes) and Share of <key> >= 500 above 5%, an error rate against all the service's records. A numeric field that is not a status offers p95 of <key> above N, with N the highest sample value rounded up. Any other string field offers a count of records where the field equals the sample value. The p95 and share options create an alert on a computed value or a percentage; see Alerts and incidents.

If the preview records share a service, the panel and alert are limited to it. You need the member role.

Good to know#

• Pipelines only affect logs that arrive after a route is active. Stored logs are not changed.
• A function that cannot read a log leaves it as it was; a pipeline never makes a batch fail.
• Pipelines apply to logs only. Traces and metrics are stored unchanged.

Next: see every parse format in Parse logs at ingest.