Parsers
A parser turns raw log lines into fields you can search, chart and alert on. You build one from your own events: LogPulse groups them into templates, you click the parts that should become fields, and the parser applies to that data wherever it comes from.
Overview
Open Configure → Data processing → Parsers in the app. The page has three tabs:
| Tab | What it is for |
|---|---|
| Build | The guided builder: source → events & templates → extract & save. The way in for most parsers. |
| Rules | Every parser rule in the order ingest tries them, with what each rule costs per event. Write or edit a rule by hand (regex, grok, JSON, logfmt, CSV). |
| Built-in (agent) | The parsers the LogPulse agent already runs on the host (sshd, sudo, nginx, firewalls, …). |
Where parsing runs
Parsing happens once, when an event is indexed, in the same step for every source: the LogPulse agent, the HTTP API, OTLP, Vector, webhooks, pipelines and cloud integrations. A parser is scoped to the data (for example sourcetype:nginx), not to how it arrived, so the same parser covers an nginx log shipped by an agent and one pushed over the API.
Extracted fields are stored with the event and can be used directly in LPQL: sourcetype=app user=alice | stats count by src_ip.
Build a parser
1. Pick a source
The builder lists what came in over the last 24 hours, by sourcetype, with how much of it is already parsed. Click one, or type any scope: sourcetype, index, source, host or service followed by a value. Wildcards work: source:app-*.
2. Events & templates
LogPulse reads up to 1,000 recent events of the source and groups them into templates: lines that share the same structure and differ only in their values. The parts that change, such as IP addresses, numbers, timestamps and IDs, are shown as coloured placeholders:
2026-10-02T10:00:00Z INFO user alice logged in from 10.0.0.5 via ssh
2026-10-02T10:01:12Z INFO user bob logged in from 10.0.0.6 via ssh<TS> INFO user <value> logged in from <IP> via sshEach template says whether a parser already handles it. Templates marked Not parsed are what is left to do; a new log format that starts arriving later shows up here as a new, unparsed template. Switch to Events to see the raw lines and which template each belongs to. Pick a shorter or longer window (last hour, 24 hours, 7 days) to change the sample.
3. Extract & save
Click Extract fields on a template, then click each coloured part that should become a field and give it a name (a suggestion is filled in). Parts you don't click still have to match, so the parser stays specific to this template, but they are not stored as fields.
The preview on the right runs the parser over the whole sample and shows the extracted fields per event and how many events it matches. Prefer to write it yourself? Write a regex instead takes a regular expression with named groups and previews it the same way.
Save parser stores it as a parser rule for the source. It applies to events that arrive from then on, within about a minute.
Event time & level
A parser can say which extracted field is the event's time and which is its level:
| Setting | Effect |
|---|---|
Event time (_time) | The value becomes the event time (_time) used by searches, time ranges and charts. Without it, the event keeps the time it arrived with. |
| Level | The value becomes the level column (error, warn, info, …). Common spellings and syslog severities 0–7 are normalised. |
The time the event was indexed is always kept separately as _indextime, so a parsed time in the past or the future never hides an event: search on what arrived with _index_earliest=-15m. See index time in LPQL.
Time formats
Leave the format empty and LogPulse recognises the common shapes: ISO 8601, epoch in seconds, milliseconds, microseconds or nanoseconds, access-log time (02/Oct/2026:10:15:30 +0200) and syslog time (Oct 2 10:15:30). For anything else, give a strptime-style format, as with Splunk's TIME_FORMAT:
| Code | Meaning | Code | Meaning |
|---|---|---|---|
%Y | Year (2026) | %y | Year, two digits |
%m | Month (01–12) | %b / %B | Month name (Oct / October) |
%d / %e | Day of month | %j | Day of year |
%H / %I %p | Hour, 24h / 12h with AM/PM | %M / %S | Minute / second |
%f, %3N, %6N, %9N | Fraction of a second | %z / %Z | Zone (+0200, +02:00, Z / UTC) |
%s | Epoch seconds | %T / %F | %H:%M:%S / %Y-%m-%d |
%d-%m-%Y %H:%M:%S,%3N 02-10-2026 10:15:30,123
%Y/%m/%d %I:%M:%S %p 2026/10/02 03:15:30 PM
%a, %d %b %Y %T %z Fri, 02 Oct 2026 10:15:30 +0000_time_parse_error field that says why.Editing & deleting
In the builder, a template that one of your parsers handles shows Edit parser. It opens the same screen with the fields, time and level filled in; change them, check the preview and save, or switch the parser off or delete it. In the Rules tab, parsers made from a template open in the builder; other rules open in the rule editor, whose Test button also shows the resulting _time and level. Deleting always asks for confirmation.
Rules & order
Under the hood a parser is one or more parser rules with the same scope, typically one per template. For each event, LogPulse tries every rule whose scope matches, highest priority first; the first rule that matches the line wins. A rule that does not match hands the line to the next one.
| Rule type | Use it for |
|---|---|
| Regex | Named groups become fields. What the builder generates. |
| Grok | Readable patterns such as %{IPORHOST:client} %{NUMBER:status}. |
| JSON | Flattens a JSON event into fields (rename them with field mappings). |
| Logfmt / Key=Value | key=value pairs. JSON and key=value are also detected automatically for events that no rule is scoped to. |
| CSV | Comma separated lines whose columns always sit in the same place, such as Palo Alto firewall logs. No regex, so it is the fastest type. |
JSON and key=value events from a source without rules are extracted automatically, so they rarely need a parser; add a rule only to rename fields or to choose the time and level field.
CSV rules
The pattern of a CSV rule is the list of column names, separated by commas. Column one gets the first name, column two the second, and so on. Leave a name empty to skip a column. Values in double quotes may contain commas and doubled quotes (RFC 4180).
,,,,log_subtype,,,src_ip,dest_ip,,,rule,src_user1,2026/11/14 10:15:30,0123,TRAFFIC,end,2561,2026/11/14 10:15:30,10.0.0.5,8.8.8.8,0.0.0.0,0.0.0.0,allow-dns,aliceThis gives log_subtype=end, src_ip=10.0.0.5, dest_ip=8.8.8.8, rule=allow-dns and src_user=alice. A line with fewer columns than the rule reads does not match, so the next rule gets it. Give the rule a scope such as sourcetype:pan:traffic so it only sees the lines it is meant for.
(?:[^,]*,){7}) is refused as unsafe and is slow on long lines. A CSV rule reads a 118 column Palo Alto line in about 2 µs.Speed per rule
Every event that comes in passes through your parser rules, so a slow rule slows down your whole ingest. LogPulse measures each rule while it runs and shows the result in the Speed column of the Rules tab, over the last 24 hours:
| Label | When |
|---|---|
| Fast | On average 50 µs per event or less. |
| Check | On average more than 50 µs per event, or the rule matched less than 20% of at least 1,000 events it was tried on. |
| Slow | On average more than 200 µs per event, or at least one event took more than 2 ms. |
| Paused | The rule was paused in the last 24 hours, see below. |
Below the label you see the time per event and how many of the events it was tried on it parsed. Hover over the cell for the slowest event, the number of events and the CPU time it used. When a rule needs attention, a notice above the list names it and says what you can do. The Key=Value Extraction row is the automatic JSON and key=value detection for events no rule parsed.
Slow rules are paused
The budget is 2 ms per event per rule. When a rule goes over it at least 20 times in a minute, and that is at least 0.5% of the events it was tried on, LogPulse pauses the rule for 5 minutes. During the pause its events go to the next rule or to automatic detection, so no event is lost; they may just have fewer fields. After 5 minutes the rule runs again by itself. It is never switched off for good.
You get an in-app notification, at most once per rule per hour, and the pause is logged to index=_internal sourcetype=ingest_errors action=parser_rule_paused.
Make a rule faster
| Tip | Why it helps |
|---|---|
| Give the rule a scope | A rule with a sourcetype, source or host only runs on the events it belongs to, instead of on everything. |
| Anchor the pattern with ^ | The regex then fails at the first character instead of trying every position in the line. |
| Avoid .* in the middle | Use [^ ]+ or [^,]* instead: it cannot run past the end of a field and back. |
| Match fixed text | A literal word such as sshd or TRAFFIC near the start rules out other lines quickly. |
| Use CSV for columns | For comma separated logs, a CSV rule reads columns by position without a regex. |
| Use JSON for JSON | A JSON rule reads an event that arrived as JSON directly, without parsing text. |
Check the result in the rule editor with Test, then watch the Speed column: it updates every minute.
Parsers from the Intelligence Hub
LogPulse writes parsers for well-known products and sends them to your workspace through the Intelligence Hub, next to the detection rules. A detection rule reads fields such as src_ip, user or action. When a product sends raw lines and no parser makes those fields, the rule can never fire. A hub parser closes that gap: the rule and the parser for its log source arrive together.
| What | How it works |
|---|---|
| One per sourcetype | Each hub parser reads exactly one sourcetype, such as pan:traffic or nginx_access. Send your data with that sourcetype and it is parsed. |
| Data model fields | Field names come from the LogPulse data model, the same names the hub detection rules read. |
| Tested | Every hub parser is tested on example lines of the product, including a line of another format that it must leave alone. |
| Fast | Each test line must parse in less than 2 ms. In your workspace a hub parser is measured like your own rules and shows in the Speed column. |
| Kept up to date | A new version reaches your workspace within 15 minutes. Every version is tested again before it goes out. |
You find them on the Rules tab under From the Intelligence Hub. Open one to see the fields it gives and an example line.
Your rules go first
Hub parsers run after all your own rules. When one of your rules parses a line, that result counts and the hub parser does not see the line. When your rule does not match a line, the hub parser gets it. So a rule of yours on the same sourcetype always wins, and the hub parser catches what it leaves.
Switch on, off or copy
Hub parsers are off until you switch them on, just like detection rules from the hub. Switch on the ones for the data you send, and off again if you parse that data your own way; the change applies within a minute. Want to change how a hub parser works? Choose Copy as own rule. The editor opens with the hub parser filled in, and your copy runs before the hub parser from then on.
Parsing on the agent
The LogPulse agent runs built-in parsers on the host for common formats: sshd, sudo and PAM, nginx, ufw/iptables, Cisco ASA, UniFi, dnsmasq, Suricata EVE JSON, BIND and Infoblox DNS, ISC dhcpd, Zscaler NSS feeds, Stormshield, Trend Micro CEF, Barracuda CloudGen Firewall, Juniper SRX, Squid, Blue Coat ProxySG and more. Their fields arrive with the event. Everything else is parsed by the parsers you build here, which apply no matter which agent or integration sent the data. See Agents: parsing.
Already stored events
A new or changed parser does not rewrite events that are already stored. To search older events the same way, use rex at search time:
sourcetype=app "logged in" | rex "user (?<user>\S+) logged in from (?<src_ip>\S+)" | stats count by userTroubleshooting
The builder shows no events
The scope matched nothing in the chosen window. Pick a longer window, check the spelling of the sourcetype, and check that your role can read the index the data lands in.
A saved parser does not match new events
Parse errors are logged per rule to index=_internal sourcetype=ingest_errors action=parser_error with a sample event. The usual cause is a log format that changed: the builder then shows the new format as a separate, unparsed template.
A rule shows Slow or Paused
Open the rule and follow the tips in Make a rule faster. Most often a rule has no scope, or its regex starts with .*. Find the pauses with index=_internal sourcetype=ingest_errors action=parser_rule_paused | stats count by rule_name.
A hub detection rule never fires on my data
Check the sourcetype of your events with | stats count by sourcetype. A hub parser only reads the sourcetype it is made for. If your data arrives with another sourcetype, set the right one at the source, or copy the hub parser as your own rule with your sourcetype as its scope.
Events land at the wrong time
Check the time field in the preview: not read means the format does not match. Without a zone the time is read as UTC; include %z when the log carries one. Compare _time with _indextime to see the shift: | eval lag=_indextime-_time | stats avg(lag) max(lag).