Data processing

Parsers

A parser turns raw log lines into fields you can search, chart and alert on. You build one from your own events: LogPulse groups them into templates, you click the parts that should become fields, and the parser applies to that data wherever it comes from.

Overview

Open Configure → Data processing → Parsers in the app. The page has three tabs:

Tab	What it is for
Build	The guided builder: source → events & templates → extract & save. The way in for most parsers.
Rules	Every parser rule in the order ingest tries them. Write or edit a rule by hand (regex, grok, JSON, logfmt).
Built-in (agent)	The parsers the LogPulse agent already runs on the host (sshd, sudo, nginx, firewalls, …).

Where parsing runs

Parsing happens once, when an event is indexed, in the same step for every source: the LogPulse agent, the HTTP API, OTLP, Vector, webhooks, pipelines and cloud integrations. A parser is scoped to the data (for example sourcetype:nginx), not to how it arrived, so the same parser covers an nginx log shipped by an agent and one pushed over the API.

Extracted fields are stored with the event and can be used directly in LPQL: sourcetype=app user=alice | stats count by src_ip.

Build a parser

1. Pick a source

The builder lists what came in over the last 24 hours, by sourcetype, with how much of it is already parsed. Click one, or type any scope: sourcetype, index, source, host or service followed by a value. Wildcards work: source:app-*.

Tip: From an agent group, the Parse step links each of the group's sourcetypes straight into the builder.

2. Events & templates

LogPulse reads up to 1,000 recent events of the source and groups them into templates: lines that share the same structure and differ only in their values. The parts that change, such as IP addresses, numbers, timestamps and IDs, are shown as coloured placeholders:

Events

2026-10-02T10:00:00Z INFO user alice logged in from 10.0.0.5 via ssh
2026-10-02T10:01:12Z INFO user bob logged in from 10.0.0.6 via ssh

Template

<TS> INFO user <value> logged in from <IP> via ssh

Each template says whether a parser already handles it. Templates marked Not parsed are what is left to do; a new log format that starts arriving later shows up here as a new, unparsed template. Switch to Events to see the raw lines and which template each belongs to. Pick a shorter or longer window (last hour, 24 hours, 7 days) to change the sample.

3. Extract & save

Click Extract fields on a template, then click each coloured part that should become a field and give it a name (a suggestion is filled in). Parts you don't click still have to match, so the parser stays specific to this template, but they are not stored as fields.

The preview on the right runs the parser over the whole sample and shows the extracted fields per event and how many events it matches. Prefer to write it yourself? Write a regex instead takes a regular expression with named groups and previews it the same way.

Save parser stores it as a parser rule for the source. It applies to events that arrive from then on, within about a minute.

Event time & level

A parser can say which extracted field is the event's time and which is its level:

Setting	Effect
`Event time (_time)`	The value becomes the event time (_time) used by searches, time ranges and charts. Without it, the event keeps the time it arrived with.
Level	The value becomes the level column (error, warn, info, …). Common spellings and syslog severities 0–7 are normalised.

The time the event was indexed is always kept separately as _indextime, so a parsed time in the past or the future never hides an event: search on what arrived with _index_earliest=-15m. See index time in LPQL.

Time formats

Leave the format empty and LogPulse recognises the common shapes: ISO 8601, epoch in seconds, milliseconds, microseconds or nanoseconds, access-log time (02/Oct/2026:10:15:30 +0200) and syslog time (Oct 2 10:15:30). For anything else, give a strptime-style format, as with Splunk's TIME_FORMAT:

Code	Meaning	Code	Meaning
`%Y`	Year (2026)	`%y`	Year, two digits
`%m`	Month (01–12)	`%b / %B`	Month name (Oct / October)
`%d / %e`	Day of month	`%j`	Day of year
`%H / %I %p`	Hour, 24h / 12h with AM/PM	`%M / %S`	Minute / second
`%f, %3N, %6N, %9N`	Fraction of a second	`%z / %Z`	Zone (+0200, +02:00, Z / UTC)
`%s`	Epoch seconds	`%T / %F`	%H:%M:%S / %Y-%m-%d

Examples

%d-%m-%Y %H:%M:%S,%3N        02-10-2026 10:15:30,123
%Y/%m/%d %I:%M:%S %p          2026/10/02 03:15:30 PM
%a, %d %b %Y %T %z            Fri, 02 Oct 2026 10:15:30 +0000

Note: A time without a zone is read as UTC. A time more than 2 days in the future or 10 years in the past is refused: the event keeps its arrival time and gets a _time_parse_error field that says why.

Editing & deleting

In the builder, a template that one of your parsers handles shows Edit parser. It opens the same screen with the fields, time and level filled in; change them, check the preview and save, or switch the parser off or delete it. In the Rules tab, parsers made from a template open in the builder; other rules open in the rule editor, whose Test button also shows the resulting _time and level. Deleting always asks for confirmation.

Rules & order

Under the hood a parser is one or more parser rules with the same scope, typically one per template. For each event, LogPulse tries every rule whose scope matches, highest priority first; the first rule that matches the line wins. A rule that does not match hands the line to the next one.

Rule type	Use it for
Regex	Named groups become fields. What the builder generates.
Grok	Readable patterns such as %{IPORHOST:client} %{NUMBER:status}.
JSON	Flattens a JSON event into fields (rename them with field mappings).
Logfmt / Key=Value	key=value pairs. JSON and key=value are also detected automatically for events that no rule is scoped to.

JSON and key=value events from a source without rules are extracted automatically, so they rarely need a parser; add a rule only to rename fields or to choose the time and level field.

Parsing on the agent

The LogPulse agent runs built-in parsers on the host for common formats: sshd, sudo and PAM, nginx, ufw/iptables, Cisco ASA, UniFi, dnsmasq and more. Their fields arrive with the event. Everything else is parsed by the parsers you build here, which apply no matter which agent or integration sent the data. See Agents: parsing.

Already stored events

A new or changed parser does not rewrite events that are already stored. To search older events the same way, use rex at search time:

LPQL

sourcetype=app "logged in" | rex "user (?<user>\S+) logged in from (?<src_ip>\S+)" | stats count by user

Troubleshooting

The builder shows no events

The scope matched nothing in the chosen window. Pick a longer window, check the spelling of the sourcetype, and check that your role can read the index the data lands in.

A saved parser does not match new events

Parse errors are logged per rule to index=_internal sourcetype=ingest_errors action=parser_error with a sample event. The usual cause is a log format that changed: the builder then shows the new format as a separate, unparsed template.

Events land at the wrong time

Check the time field in the preview: not read means the format does not match. Without a zone the time is read as UTC; include %z when the log carries one. Compare _time with _indextime to see the shift: | eval lag=_indextime-_time | stats avg(lag) max(lag).