Data processing

Raw log replay

LogPulse keeps the original logs of the last 60 minutes, exactly as they came in. Replay them to build and test a pipeline on real data before it touches a single stored event.

Overview

Every log LogPulse accepts first lands in a durable ingest buffer (see Ingest priority). The buffer keeps the logs as they arrived: before parser rules and before any pipeline. So the replay also shows logs that no pipeline has ever touched, such as a new source you are about to route.

	Search	Raw replay
What you see	Stored events, after parsing and pipelines	The original logs, as they came in
How far back	Your full retention	The last 60 minutes
Dropped by a pipeline	Not there	Still there
Used for	Searching, alerts, dashboards	Building and testing pipelines

Test a pipeline

Raw logs show up in two places in the stream pipeline editor. In the test dialog, pick real logs from Raw logs from the last hour and run the test: you see per step how many events went in and out, what was dropped and which errors occurred, before you promote. In the start panel, Pick from raw logs fills in which logs the pipeline should take (index, sourcetype, source, host) from a log you click. See Building pipelines.

Filters

Filter	Matches
Index, sourcetype, source, host	The same patterns as a stream pipeline match, up to 20 per field
Level	Exact, case-insensitive (`error`, `warn`, …)
API key	Logs sent with that key, or by that agent
Text	Every word must appear in the log

Next to the filters you see the top values for index, sourcetype, source, host, level and API key in the chosen span, so you know what there is to filter on.

The sample

A replay returns at most 500 logs (200 by default), spread evenly over the time span rather than only the newest ones. The logs are parsed with your organization's parser rules, so the fields look like they will after ingest. The result also says how many logs matched, how much of the span was read, and whether the sample was cut off.

Access and audit

Raw logs have not been through your masking pipelines yet. That is why a replay needs both logs:read and pipelines:update, and why every replay is written to the audit log with its time span, filters and number of logs.

How far back

The replay goes back 60 minutes. On a dedicated environment the window can be made longer, up to seven days; ask us.

Note: An environment that does not keep raw logs shows this in the pipeline editor, and the API answers 409 with REPLAY_UNAVAILABLE.

API

Call	What it does
`GET /api/v1/stream-replay`	Whether replay is available, and `maxMinutes`
`POST /api/v1/stream-replay/raw`	Replay: `minutes`, optional `limit` (1 to 500) and `filters`

Example

curl -X POST https://api.logpulse.io/api/v1/stream-replay/raw \
  -H "Authorization: Bearer $LOGPULSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"minutes": 30, "limit": 100, "filters": {"sourcetype": ["nginx:access"], "text": "POST /login"}}'