Building Pipelines
LogPulse Pipelines let you build data and response workflows in a visual drag-and-drop editor. 27 node types in 7 categories fetch data from external sources, transform and enrich it, control the flow, load results into LogPulse and run response actions. Most steps need no code; JSONata and JavaScript are there when you need them.
Overview
The editor is a visual canvas built on React Flow. Drag steps from the palette, connect them and configure each one in its panel. A pipeline is a data, action or stream pipeline; the kind decides which steps you can use and what can start it.
Visual Drag & Drop
Build pipelines visually by dragging nodes from the palette and connecting them with edges.
27 Node Types
Extract, transform, route, load and respond with purpose-built steps in 7 categories.
Real-Time Execution
Watch pipelines execute in real-time with per-node status, timing, and payload inspection.
Version History
You edit a draft. Promoting it creates the version that runs, and you can roll back to an earlier version at any time.
Getting Started
Follow these steps to create your first pipeline:
Create a Pipeline
Navigate to Pipelines in the sidebar and click "New Pipeline". Give it a name and optional description.
Add a Start Node
A start node is automatically added. This is the entry point of your pipeline.
Build Your Flow
Drag nodes from the palette on the left and connect them by drawing edges between handles.
Configure Nodes
Click any node to open its configuration panel. Set URLs, mappings, conditions, and more.
Test & Activate
Run a test with a test event and inspect each step. Then promote the draft and switch the pipeline to active.
Pipeline Kinds
You pick the kind when you create a pipeline. The kind decides which nodes you can use and what can start the pipeline.
| Kind | What it is for | Runs | Only this kind has |
|---|---|---|---|
| data | Fetch, transform and write data into an index (ETL) | Schedule, webhook or API key, manually, from another pipeline | — |
| action | SOAR responses: approvals and response actions | Like data, plus alerts, KPIs, SLOs, health checks and notables | waitForApproval, responseAction |
| stream | Shape events inside ingest, between the parser rules and storage | Once per event, with no runs or run history | streamSample, streamDedupe, streamDrop, streamSink |
A stream pipeline only allows steps without side effects and without waiting: start, transform, condition, redactMask and the four stream nodes. Instead of a run history it shows counters and samples per step.
Node Categories
Nodes are organized into 7 color-coded categories in the palette. Which nodes you can use depends on the kind of pipeline.
| Category | Color | Node Types | Description |
|---|---|---|---|
| Trigger | Emerald | start | Entry point of every pipeline |
| Extract | Blue | httpRequest, splunkSearch, lpqlSearch | Pull data from external APIs, Splunk or LogPulse itself |
| Transform | Violet | transform, json, csv, redactMask, mapCommonSchema, lookup, aiAgent | Modify, parse, mask, map and enrich data |
| Flow Control | Amber | condition, split, join, wait, callPipeline, streamSample*, streamDedupe* | Branch, fan out and recombine, pause, or hand off to another pipeline |
| Load | Cyan | logpulseIngest, upsertEntities, respondWebhook, streamDrop*, streamSink* | Write events or entities, answer a webhook, or end a stream path |
| Utility | Slate | log, error | Debug logging and stopping a run with an error |
| Respond | Rose | waitForApproval†, responseAction† | Approval gates and SOAR response actions |
* Stream pipelines only. † Action pipelines only.
Drag any node from the palette onto the canvas to add it. Connect nodes by clicking an output handle (right side) and dragging to an input handle (left side).
Trigger Nodes
Every pipeline starts with a trigger node that defines how and when the pipeline executes.
The start node is the entry point and holds the test event for test runs. Every pipeline has exactly one. How the pipeline starts is set in the pipeline settings, not on the node.
A pipeline can be started in these ways:
| Mode | Description | Configuration |
|---|---|---|
| Scheduled | Runs on a cron schedule; the input is empty, so the pipeline fetches its own data | Cron expression (5 or 6 fields) + timezone, default UTC |
| HTTP ingestion | Runs on every request to its webhook URL or on events sent with a linked API key; the posted body is the input | Webhook token, or link the pipeline to an API key |
| Manual | Run now in the editor or through the API | A promoted version |
| Alerts and automation | Action pipelines only: started by an alert, KPI, SLO or health check route, or by an automation trigger on notables | A notification route or automation trigger that points at the pipeline |
| Other pipelines | A Call Pipeline step or a run_pipeline response action | The calling pipeline |
Extract Nodes
Extract nodes pull data from external sources into your pipeline. They support dynamic configuration, authentication, and retry logic.
HTTP Request
Make HTTP requests to any external API. Supports all standard methods, custom headers, query parameters, request bodies, and advanced retry/redirect handling. All string fields accept Dynamic Values for runtime interpolation.
| Setting | Type | Description |
|---|---|---|
| URL | DynamicValue | Target URL for the request |
| Method | DynamicValue | GET, POST, PUT, PATCH, or DELETE |
| Headers | Key-Value pairs | Custom request headers (DynamicValue per header) |
| Query Params | Key-Value pairs | URL query parameters (DynamicValue per param) |
| Body | DynamicValue | Request body (for POST/PUT/PATCH) |
| Timeout | Number | Request timeout in ms (default: 30000) |
| Retries | Number | Number of retry attempts on failure |
| Retry Delay | Number | Delay between retries in ms |
| Follow Redirects | Boolean | Follow HTTP redirects (default: true) |
| Max Redirects | Number | Maximum redirects to follow (default: 5) |
| Validate SSL | Boolean | Validate SSL certificates (default: true) |
| Output Path | String | Where to store the response in the payload |
Splunk Search
Execute SPL (Search Processing Language) queries against a Splunk instance and pull results into your pipeline. Useful for migrating data from Splunk or building cross-platform workflows.
| Setting | Type | Description |
|---|---|---|
| Host | DynamicValue | Splunk server hostname |
| Port | DynamicValue | Splunk management port (default: 8089) |
| Protocol | https | http | Connection protocol (default: https) |
| Auth Type | basic | token | Authentication method |
| Query | DynamicValue | SPL search query string |
| Earliest Time | String | Search time range start (e.g., -24h) |
| Latest Time | String | Search time range end (e.g., now) |
| Max Results | Number | Maximum result count (default: 1000) |
| Timeout | Number | Search timeout in ms |
| Output Path | String | Where to store search results |
LogPulse Search
Run an LPQL query against your own organization, with the same engine as the search page and your namespace permissions. The rows land in the message for the next steps.
| Setting | Description |
|---|---|
| Query | LPQL query (supports dynamic values) |
| Earliest Time | Default start of the time range, e.g. -24h. Ignored when the query has its own earliest= |
| Latest Time | Optional end of the time range, e.g. now. Ignored when the query has its own latest= |
| Max Results | Maximum number of rows |
| Output Path | Where the rows land in the payload (default: payload) |
Authentication
Both HTTP Request and Splunk Search nodes support multiple authentication methods. Credentials can use Dynamic Values to reference environment variables securely.
| Auth Type | Fields | Description |
|---|---|---|
| None | — | No authentication |
| Basic | Username, Password | HTTP Basic authentication (Base64 encoded) |
| Bearer | Token | Bearer token in Authorization header |
| API Key | Header Name, Value | Custom header with API key (default: X-API-Key) |
| Multi-Step | Steps[], Token Rules | Sequential auth flow for OAuth/complex scenarios |
Multi-step authentication enables complex OAuth and token-based flows. Define sequential steps that extract values from responses and use them in subsequent requests:
Step 1: POST /auth/login
Body: { "username": "{{envVar:API_USER}}", "password": "{{envVar:API_PASS}}" }
Extract: body → $.access_token → accessToken
Step 2: GET /api/data
Header: Authorization = Bearer {{accessToken}}
Extract: body → $.results → resultsEach auth step can extract values from three sources:
| Source | Description | Example |
|---|---|---|
| body | Extract from JSON response body | $.access_token |
| header | Extract from response header | X-Session-Id |
| cookie | Extract from response cookie | session_cookie |
Transform Nodes
Transform nodes modify, parse, enrich, and reshape data as it flows through the pipeline. They range from simple field operations to advanced JavaScript transformations and schema mapping.
Transform
The general-purpose transform node supports both a simple mapping mode and an advanced JavaScript mode for complex transformations.
Simple Mode
In simple mode the node runs a list of operations in order; later operations see what earlier ones changed. Values can come from these sources:
| Value Type | Description | Example |
|---|---|---|
| string | Text, with {{ }} templates | "{{ payload.user.name }}" |
| number | Numeric value | 42 |
| boolean | True or false | true |
| jsonata | A JSONata expression on the message | $count(payload.items) |
| message | Copy the value at a path in the message | payload.user.id |
| pipelineContext / tenantContext | Read a key from the context store | cursor.lastSeenId |
| variable | A pipeline or organization variable (older form) | ${pipeline.variables.api_url} |
| expression | Sandboxed JavaScript (older form) | data.level.toUpperCase() |
Advanced Mode (JavaScript)
In advanced mode, write JavaScript that receives the input payload and returns the result. It runs in a sandbox, and data, payload, input and $ all point to the input payload.
// Variables: data, payload, $ (all reference input payload)
const enriched = {
...data,
processed: true,
timestamp: new Date().toISOString(),
severity: data.level === 'error' ? 'high' : 'normal',
tags: [...(data.tags || []), 'processed']
};
return enriched;Transform Operations
In simple mode the Transform node runs these operations in order:
| Operation | Fields | Description |
|---|---|---|
| set | target, value, scope | Write a value to a path. Scope message (default), pipelineContext or tenantContext |
| delete | targets[], scope | Remove one or more paths |
| replace | target, from, to | Replace text inside a string field |
| move | from, to | Move (rename) a field |
| keep | targets[] | Keep only these paths and drop everything else |
| coerce | target, targetType | Convert to string, number, boolean or array |
| extract | source, target, mode | Parse a string field with a regex, a delimiter (split) or key=value pairs (kv) |
Operation 1: set → target: "env", value: "production" (string)
Operation 2: move → from: "msg", to: "message"
Operation 3: delete → targets: ["debug_info"]
Operation 4: coerce → target: "status_code", targetType: numberJSON & CSV Parsing
Dedicated nodes for parsing and formatting structured data formats. Use these when you need to convert between strings and structured objects.
JSON Node
| Mode | Description |
|---|---|
| parse | Convert a JSON string to an object. Specify input path (source string) and output path. |
| stringify | Convert an object to a JSON string. Specify input path (source object) and output path. |
CSV Node
| Setting | Description |
|---|---|
| Mode | parse (CSV → array of objects) or format (objects → CSV string) |
| Delimiter | Field separator character (default: comma) |
| Has Headers | First row contains field names (default: true) |
| Input Path | Path to the CSV string or array in payload |
| Output Path | Where to store the parsed/formatted result |
Redact / Mask
Automatically detect and mask personally identifiable information (PII) and sensitive data before it reaches storage. Supports 8 built-in detection patterns and custom regex rules.
Built-in Patterns
| Pattern | Default Action | Example |
|---|---|---|
| Replace → [EMAIL] | [email protected] → [EMAIL] | |
| Phone | Replace → [PHONE] | +1-555-0123 → [PHONE] |
| Credit Card | Mask (keep last 4) | 4111-1111-1111-1234 → ****-****-****-1234 |
| SSN | Replace → ***-**-**** | 123-45-6789 → ***-**-**** |
| IP Address | Replace → [IP] | 192.168.1.1 → [IP] |
| JWT Token | Replace → [JWT] | eyJhbGciOi... → [JWT] |
| API Key | Replace → [API_KEY] | sk-abc123... → [API_KEY] |
| IBAN | Mask (keep first 4) | NL91ABNA0417164300 → NL91************** |
Redaction Actions
| Action | Description | Options |
|---|---|---|
| mask | Replace characters with mask character | maskChar (default: *), style: fixed / keepLastN / keepFirstN, keepN |
| hash | SHA-256 hash of matched value | Irreversible, consistent hash |
| remove | Delete the field entirely | — |
| replace | Replace with custom text | Custom replacement string |
Custom Rules
Define your own detection rules using regex patterns or field path matching for data types not covered by the built-in presets.
| Match Type | Description | Example |
|---|---|---|
| regex | Regular expression pattern matching | /\b[A-Z]{2}\d{6}\b/ (passport numbers) |
| fieldPath | Match specific field paths | user.email, request.headers.authorization |
Schema Mapping
Map incoming data to standard or custom schemas. This ensures consistent field naming across different data sources and enables compatibility with downstream tools.
Schema Templates
| Template | Description |
|---|---|
| ECS | Elastic Common Schema: standard field naming for Elasticsearch/OpenSearch |
| CIM | Common Information Model: normalized event field schema |
| OTEL | OpenTelemetry: vendor-neutral observability standard |
| LogPulse | Native LogPulse data models: custom models defined in your organization |
| Custom | User-defined mappings without a base template |
The mapping mode determines how missing fields are handled:
| Mode | Description |
|---|---|
| Permissive | Missing source fields are skipped. Partial output is produced without errors. |
| Strict | All required fields must be present. Pipeline errors if required fields are missing. |
Fields can be coerced to specific types during mapping:
| Type | Description |
|---|---|
| string | Convert to string |
| int | Convert to integer |
| float | Convert to floating-point number |
| boolean | Convert to boolean |
Lookup
Enrich events by matching a field value against a lookup dataset. This is useful for adding context like hostname-to-team mappings, IP geolocation data, or CMDB asset information.
| Setting | Description |
|---|---|
| Dataset | Select a lookup dataset from your organization |
| Input Key Path | Dot-notation path to the lookup key in your payload |
| Lookup Key Field | Field name in the dataset to match against |
| Output Mode | How to output matched data (see below) |
| Field Mappings | Optional: map specific lookup fields to event fields |
Output Modes
| Mode | Description |
|---|---|
| mergeIntoEvent | Merge all lookup fields into the event payload |
| writeToTargetPath | Write lookup data to a specific path in the payload |
| embedLookup | Embed the full lookup record as a nested object |
AI Agent
A model reads the message and your instructions, may call the LogPulse tools you allow, and answers as text or as JSON that matches a schema. It runs as the person who published the running version and uses AI credits, also in test runs.
| Setting | Description |
|---|---|
| Instructions | What the model should do; {{ payload.x }} reads the message |
| Tools | LogPulse tools the model may call. Tools that propose a change only in action pipelines |
| Output | text, or json with an output schema |
| Output Path | Where the answer lands in the message (default: ai) |
| Max Tool Calls | Most tool calls in one step (default 5, at most 10) |
Flow Control
Flow control nodes branch, fan out and recombine, pause a run, or hand the message to another pipeline.
Condition
Route data to different branches based on field values. Each condition creates its own output handle. A message goes to every condition that matches, or to the default branch when none match.
Operators
| Operator | Symbol | Description | Value Required |
|---|---|---|---|
| eq | == | Equals | Yes |
| neq | != | Not equals | Yes |
| gt | > | Greater than | Yes |
| lt | < | Less than | Yes |
| gte | >= | Greater than or equal | Yes |
| lte | <= | Less than or equal | Yes |
| contains | contains | Contains substring | Yes |
| not_contains | not contains | Does not contain | Yes |
| regex | regex | Matches regular expression | Yes |
| exists | exists | Field exists in payload | No |
| not_exists | not exists | Field does not exist | No |
You can define several conditions, each with its own output and label. The default branch catches the messages that match none of them.
Advanced Mode (JavaScript)
In advanced mode, write JavaScript that returns an array of branch IDs (strings) or a single string. The event will be routed to the specified branch(es).
// Must return an array of branch IDs or a single string
if (data.level === 'error' && data.status_code >= 500) {
return ['critical-errors'];
} else if (data.level === 'error') {
return ['errors'];
}
return ['default'];Split & Join
Fan an array out into one message per element with split, then optionally re-merge them with join. Each fork carries a parts.id correlation marker so a downstream join knows which forks belong together, the same pattern Node-RED uses.
| Split Setting | Description |
|---|---|
| Array Path | Dot-notation path to the array to fan out (e.g., payload.items) |
| Output Path | Where each element lands inside the per-fork message (default: payload) |
| Concurrency | Most forks running at once. Empty: all at once; 1 processes the items one after another (a loop) |
| Rate Limit | Optional throttle: release N forks per second instead of all at once |
| Continue on Error | Log a failing fork instead of failing the split |
| Join Setting | Description |
|---|---|
| Mode | "auto" matches an upstream split via parts.id; "manual" waits for a fixed count |
| Output Path | Where the joined array is placed (default: payload) |
| Count | Required in manual mode: number of messages to collect |
| Timeout (ms) | Maximum wait for missing parts (default: 60 seconds) |
| When Incomplete | partial (default) continues with the parts that arrived; fail fails the run |
Splits and joins nest. When a split runs while parts metadata is already present, the existing parts is pushed onto a stack, and each join pops the stack on completion, so split → ... → split → ... → join → join unwinds cleanly.
Wait
Pause for a number of seconds, or until a moment (an ISO timestamp, a Unix time or a {{ payload.x }} template). A wait of more than a minute parks the run with status waiting; it resumes by itself, up to 30 days later. Test runs skip the wait.
Call Pipeline
Run the published version of another pipeline with this message, inside the same run. Its result lands at the output path (default: result). The called pipeline must finish in one go, calls nest at most 3 deep, and a data pipeline cannot call an action pipeline.
Load Nodes
Load nodes end a path: they write events to an index, update entities or answer the webhook caller.
LogPulse Ingest
Write processed events into LogPulse. Set the source type, target index and field mappings to control how events appear in search.
| Setting | Default | Description |
|---|---|---|
| Source Type | etl_pipeline | Value for the sourcetype field in LogPulse |
| Index | main | Target index for ingested events |
| Host | logpulse-etl | Host name to tag events with |
| Source | etl:<pipeline id> | Source identifier |
| Message Path | payload | What is written: an array becomes one event per element, an object one event |
| Timestamp Path | — | Path to the timestamp field |
| Level Path | — | Path to the log level field |
| Field Mapping | — | Optional custom field name mappings |
Upsert Entities
Create or update entities (such as users and hosts) from a sync, for example the members of a Cloudflare account. Each item is { key, display_name?, roles?, attributes? }; roles are stored per source, so a sync only owns and revokes its own roles. Detections read them with | lookup identities.
| Setting | Description |
|---|---|
| Items Path | Path to the array of items (default: payload) |
| Entity Type | Entity type to write (default: user) |
| Role Source | Short source name, such as cloudflare (required) |
| Prune Missing | Remove this source's roles from entities that are no longer listed |
| Total Count Path | Path to the source's total count; pruning is skipped when a run only saw part of it |
A test run only shows a preview and writes no entities.
Respond to Webhook
The answer for a webhook caller that waits for the result (?wait=N or the header Prefer: wait=N, at most 30 seconds). Set the status code (default 200), the body (the message or a {{ payload }} template) and headers. Without this step a waiting caller gets the run's output.
Utility Nodes
Utility nodes help with debugging and error handling during pipeline development and execution.
Write a debug line to the run log, from the whole message, a path or a template. Optionally also to the pipeline_logs index. A log step ends its path.
Stop pipeline execution immediately and record an error. Use this in condition branches to halt processing when invalid data is detected.
Retries & Errors
Every working step (not start, split, join, wait, approval gates, log or error) can retry and decide what happens when it fails.
| On error | What happens |
|---|---|
| stop (default) | The run fails at this step |
| continue | The next steps run with the input plus an error field |
| branch | Only the error output of the step runs, with the error on the message. Use it for "notify me when the API is down" |
Retries take a number of attempts, a delay and a backoff. Under Settings → Runs you also set the most runs at once (extra runs queue or are skipped), an idempotency key so a repeated webhook delivery starts no second run, and an error pipeline that runs after the last retry has failed.
Respond Nodes
Respond nodes only exist in action pipelines.
| Node | What it does |
|---|---|
| waitForApproval | Pause the run (status waiting_approval) until someone approves it in the Respond tab. With an expiry in hours it ends by itself as rejected (default) or approved; a rejection follows the rejected output when there is one |
| responseAction | Run a response action: notify, create_ticket, block_ip, disable_user, isolate_host or run_pipeline. In a test run it only does a dry run |
Stream Nodes
Stream nodes only exist in stream pipelines, which run once per event inside ingest. A failing step lets the event through unchanged. Test a stream pipeline on the original logs of the last 60 minutes with raw log replay.
| Node | What it does |
|---|---|
| streamSample | Keep one in N events. With a key field the same key is always kept or always dropped |
| streamDedupe | Drop an event whose key fields were already seen within a time window |
| streamDrop | End the path without storing the event; the reason shows in the counters |
| streamSink | Send the event to an external destination, and optionally store it as well |
Dynamic Values
Text fields in step settings take templates instead of fixed values. {{ payload.user.name }} reads a field and any JSONata expression works too. A value that is a single {{ }} keeps its type.
| Type | Syntax | Description | Example |
|---|---|---|---|
| Field | {{ path }} | A value from the message | {{ payload.user.id }} |
| JSONata | {{ expression }} | Any JSONata expression on the message | {{ $count(payload.items) }} |
| Variables | {{ $vars.name }}, {{ $tenantVars.name }} | Pipeline and organization variables | {{ $vars.api_url }} |
| Context | {{ $context.key }}, {{ $tenantContext.key }} | Values from the context store | {{ $context.cursor }} |
| System | {{ $system.run_id }} | Run details: run_id, pipeline_id, run_started_at and more | {{ $system.run_started_at }} |
| Secrets | {{ $secrets.name }}, {{ $tenantSecrets.name }} | Pipeline and organization secrets; only in HTTP fields and response action parameters | {{ $secrets.api_key }} |
The older placeholders, such as ${pipeline.variables.X} and ${pipeline.secrets.X}, still work.
Context Store
Pipelines are stateless by default: every run starts fresh. The Context Store is the escape hatch for state that must survive across runs: incremental cursors, dedup caches, last-seen IDs, accumulated counters, or anything where 'what did the previous run leave behind' matters. Two scopes are available: pipelineContext (private to one pipeline) and tenantContext (shared across every pipeline in your organization).
| Scope | Visibility | Typical use |
|---|---|---|
| pipelineContext | Private to one pipeline | Polling cursors, dedup caches, last-seen IDs |
| tenantContext | Shared across every pipeline in the org | Cross-pipeline coordination, shared lookup lists |
Pipeline Context
pipelineContext is bound to one pipeline. Use it for pipeline-local cursors, dedup caches, and any state you don't want to share. Values are JSON: strings, numbers, booleans, arrays, and objects all work. Object-typed values can be addressed with dot-paths on both read and write (writing to cursor.lastSeenId creates the cursor object automatically if it didn't exist).
Reading a value
Use a transform set operation with value type pipelineContext. The value field is the key (or dot-path into an object value).
Operation: set
target: payload.lastSeenId
value:
type: pipelineContext
value: cursor.lastSeenIdWriting a value
Use a transform set operation with scope pipelineContext. The target is the key (or dot-path); the value is whatever JSONata or expression you want to persist.
Operation: set
scope: pipelineContext
target: cursor.lastSeenId
value:
type: jsonata
value: $max(payload.id)Tenant Context
tenantContext is shared across every pipeline in your organization. Use it when several pipelines need to coordinate: one pipeline writes a list of active customers, another pipeline reads it. Both pipelines see the same store; consumers always read the freshest persisted value.
Operation: set
scope: tenantContext
target: active_customers
value:
type: jsonata
value: payload.customersOperation: set
target: payload.customers
value:
type: tenantContext
value: active_customersCommon Patterns
Two patterns cover most real use cases. JSONata expressions can also read the snapshot directly via the $context and $tenantContext bindings, which is convenient when you want to mix the existing state into a computation.
Incremental polling cursor
The most common use of the context store: remember where the previous run stopped so the next run only fetches new data. The first run sees no cursor, so use $exists() to provide a fallback timestamp. Advance the cursor only after a successful ingest; failed runs leave it unchanged so the next attempt re-pulls the same window.
Operation: set
target: payload.since
value:
type: jsonata
value: $exists($context.cursor.lastPolledAt)
? $context.cursor.lastPolledAt
: "1970-01-01T00:00:00Z"Operation: set
scope: pipelineContext
target: cursor.lastPolledAt
value:
type: jsonata
value: $max(payload.timestamp)Dedup cache
Skip records you've already processed by keeping a list of seen IDs in the context store. Filter the new payload to drop ids already in $context.seenIds, ingest, then append the new ids to the cache.
Operation: set
scope: pipelineContext
target: seenIds
value:
type: jsonata
value: $append($context.seenIds, payload.id)Pipeline Settings
Pipeline-level settings control how and when your pipeline runs, as well as debugging and testing features.
Trigger Modes
Set how the pipeline starts. Schedules and webhooks read these settings from the production version.
Scheduled Execution
Run the pipeline on a cron schedule in the schedule's timezone (default UTC). It runs only when the pipeline is active and promoted and the schedule is enabled. A scheduled run starts with an empty input, so the pipeline fetches its own data.
| Setting | Description |
|---|---|
| What starts it | On the Start step or the Settings tab: on a schedule, when data comes in, or only when asked. Picking "On a schedule" switches the schedule on |
| Cron Expression | 5- or 6-field cron (e.g., */5 * * * * for every 5 minutes). Shortcuts such as @hourly are not accepted |
| Timezone | Timezone for the cron schedule (default: UTC) |
| Publish | What starts a pipeline is part of the published version. The Start step says what the live version does and what is set but not published yet |
*/5 * * * * Every 5 minutes
0 * * * * Every hour
0 9 * * 1-5 Weekdays at 9:00 AM
0 0 * * * Daily at midnight
0 */6 * * * Every 6 hoursHTTP Ingestion
Send events through the pipeline with its webhook URL (a long token in the URL that you can renew and limit to IP lists) or with an API key linked to the pipeline. The posted body is the input. A caller that waits (?wait=N or Prefer: wait=N, at most 30 seconds) gets the answer of a Respond to Webhook step, or the run's output.
Active / Inactive
The Active toggle is a master switch. A paused pipeline is skipped everywhere: schedule, webhook, API key, alerts and automations. You can still edit it and run tests.
Test Events
Test events are sample payloads that you save on the pipeline. A test run executes the draft with one of them, without changing production.
| Feature | Description |
|---|---|
| Create Test Event | Define a name and JSON payload on the Start step or the Settings tab. An action pipeline also offers ready-made examples of what an alert, KPI, SLO, health check or notable sends |
| Select & Run | Pick a test event on the Start step; Test starts the draft with it |
| View Results | Inspect input/output payloads at each node after a test run |
| In a test run | What happens |
|---|---|
| LogPulse Ingest | Writes nothing; shows __ingestPreview and __ingestCount |
| Response Action | Does not fire; returns a dry run |
| Upsert Entities | Writes nothing; shows a preview |
| HTTP Request | Runs for real: the call is made |
| Log | Writes to the pipeline_logs index when "log to index" is on |
| Wait | Is skipped |
Execution & Tracking
Every run is recorded with its status, the version it ran and the input and output of each step. The editor shows progress live over a WebSocket.
Run Status
| Status | Description |
|---|---|
| pending | The run is created and about to start |
| queued | Waiting for a free slot under the most runs at once |
| running | The pipeline is running |
| waiting | Parked by a Wait step; resumes by itself |
| waiting_approval | Waiting for someone to approve in the Respond tab |
| success | Finished without errors |
| failed | Stopped with an error; can be retried or resumed from the failed step |
| dead-letter | Failed after its last retry |
| cancelled | Stopped, or skipped because another run was busy |
Node Execution Details
Each node in a run is tracked individually, allowing you to pinpoint exactly where failures occurred and inspect the data at each step.
| Tracked Data | Description |
|---|---|
| Status | pending, running, success, failed, or skipped |
| Duration | Execution time in milliseconds |
| Input Payload | Data received by the node |
| Output Payload | Data produced by the node |
| Error Details | Error message (if failed) |
Real-Time Updates
The pipeline editor connects via WebSocket to receive live updates during execution. Nodes light up as they execute, showing status, duration, and errors in real-time.
| Event | Description |
|---|---|
| pipeline:run:started | Pipeline execution has begun |
| pipeline:run:finished | Pipeline completed successfully |
| pipeline:run:failed | Pipeline failed with an error |
| pipeline:node:started | A node has started executing |
| pipeline:node:finished | A node has completed |
| pipeline:node:failed | A node has failed |
| pipeline:log:emit | A Log step wrote a line |
Versioning
You always edit a draft, and test runs use the draft. Schedules, webhooks and triggers run the production version. Promoting saves the draft as a new version and makes it production.
| Feature | Description |
|---|---|
| Draft | What you edit and what a test run uses. Saving changes only the draft |
| Promote | Saves the draft as a new version with a message and makes it production |
| Production | The version that schedules, webhooks and triggers run |
| Rollback | Points production at an earlier version; the draft stays as it is |
| Load version | Opens an earlier version as the draft to continue from it |
| Pinned runs | A run keeps the version it started with, also when you promote during the run |
API Reference
All pipeline operations are available via the REST API. All endpoints require authentication and are scoped to the current organization.
Pipeline Endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/v1/pipelines | List the pipelines with their last run |
| POST | /api/v1/pipelines | Create a pipeline |
| GET | /api/v1/pipelines/:id | A pipeline with its graph, production version and draft state |
| PUT | /api/v1/pipelines/:id | Update the draft and settings (including test events) |
| DELETE | /api/v1/pipelines/:id | Delete a pipeline |
| PUT | /api/v1/pipelines/:id/active | Pause or resume a pipeline |
| POST | /api/v1/pipelines/:id/validate | Check the graph without saving |
Execution Endpoints
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/v1/pipelines/:id/test-run | Test the draft with a payload |
| POST | /api/v1/pipelines/:id/run | Run the production version now |
| GET | /api/v1/pipelines/:id/runs | Runs of a pipeline, newest first, with a per-node breakdown |
| GET | /api/v1/pipelines/:id/runs/:runId | One run with its node executions |
| POST | /api/v1/pipelines/:id/runs/:runId/retry | Run a failed run again |
| POST | /api/v1/pipelines/:id/runs/:runId/resume | Resume a failed run from its failed step |
| POST | /api/v1/pipelines/webhook/:token | Start a pipeline through its webhook URL |
Version Endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/v1/pipelines/:id/versions | The promoted versions, newest first |
| GET | /api/v1/pipelines/:id/versions/:versionId | One version with its graph |
| POST | /api/v1/pipelines/:id/promote | Promote the draft to a new production version |
| POST | /api/v1/pipelines/:id/rollback | Point production at an earlier version |
| POST | /api/v1/pipelines/:id/load-version/:versionId | Load an earlier version into the draft |