Building Pipelines

Make HTTP requests to any external API. Supports all standard methods, custom headers, query parameters, request bodies, and advanced retry/redirect handling. All string fields accept Dynamic Values for runtime interpolation.

Execute SPL (Search Processing Language) queries against a Splunk instance and pull results into your pipeline. Useful for migrating data from Splunk or building cross-platform workflows.

Both HTTP Request and Splunk Search nodes support multiple authentication methods. Credentials can use Dynamic Values to reference environment variables securely.

Multi-step authentication enables complex OAuth and token-based flows. Define sequential steps that extract values from responses and use them in subsequent requests:

Step 1: POST /auth/login
  Body: { "username": "{{envVar:API_USER}}", "password": "{{envVar:API_PASS}}" }
  Extract: body → $.access_token → accessToken

Step 2: GET /api/data
  Header: Authorization = Bearer {{accessToken}}
  Extract: body → $.results → results

Each auth step can extract values from three sources:

The general-purpose transform node supports both a simple mapping mode and an advanced JavaScript mode for complex transformations.

Simple Mode

In simple mode, define field mappings with a target key and value. Each mapping supports four value types:

Advanced Mode (JavaScript)

In advanced mode, write JavaScript code that receives the input payload and returns the transformed result. Three variables are available that all reference the input payload: data, payload, and $ (shorthand).

// Variables: data, payload, $ (all reference input payload)
const enriched = {
  ...data,
  processed: true,
  timestamp: new Date().toISOString(),
  severity: data.level === 'error' ? 'high' : 'normal',
  tags: [...(data.tags || []), 'processed']
};
return enriched;

Apply structured field-level operations without writing code. Supports five operation types that can be chained in sequence.

Operation 1: add    → field: "env", value: "production", type: string
Operation 2: rename → field: "msg", newName: "message"
Operation 3: remove → field: "debug_info"
Operation 4: coerce → field: "status_code", targetType: number

Dedicated nodes for parsing and formatting structured data formats. Use these when you need to convert between strings and structured objects.

JSON Node

CSV Node

Automatically detect and mask personally identifiable information (PII) and sensitive data before it reaches storage. Supports 8 built-in detection patterns and custom regex rules.

Built-in Patterns

Redaction Actions

Custom Rules

Define your own detection rules using regex patterns or field path matching for data types not covered by the built-in presets.

Map incoming data to standard or custom schemas. This ensures consistent field naming across different data sources and enables compatibility with downstream tools.

Schema Templates

The mapping mode determines how missing fields are handled:

Fields can be coerced to specific types during mapping:

Enrich events by matching a field value against a lookup dataset. This is useful for adding context like hostname-to-team mappings, IP geolocation data, or CMDB asset information.

Output Modes

Route data to different branches based on field values. Each condition creates a separate output handle on the node. Events are routed to the first matching condition, or to the 'else' branch if none match.

Operators

Multiple conditions can be defined, each with its own output handle and label. The 'else' branch catches all events that don't match any condition.

Advanced Mode (JavaScript)

In advanced mode, write JavaScript that returns an array of branch IDs (strings) or a single string. The event will be routed to the specified branch(es).

// Must return an array of branch IDs or a single string
if (data.level === 'error' && data.status_code >= 500) {
  return ['critical-errors'];
} else if (data.level === 'error') {
  return ['errors'];
}
return ['default'];

Iterate over arrays in the payload. The loop node processes each array element through the nodes between loopStart and loopEnd, making the current item available as a variable.

Loop-related nodes work together:

Send processed events to LogPulse for indexing. Configure the source type, target index, and field mappings to control how events appear in search results.

Configure how your pipeline is triggered. Each pipeline supports one active trigger mode.

Scheduled Execution

Run the pipeline automatically on a cron schedule. The pipeline worker checks schedules approximately every minute. The pipeline must be active and scheduled runs must be enabled.

*/5 * * * *      Every 5 minutes
0 * * * *        Every hour
0 9 * * 1-5      Weekdays at 9:00 AM
0 0 * * *        Daily at midnight
0 */6 * * *      Every 6 hours

HTTP Ingestion

Link the pipeline to an API key. Events sent via the API key are automatically routed through the pipeline before being ingested. This is ideal for processing incoming log data in real-time.

Active / Inactive

The Active toggle is a master switch. When inactive, all triggers (scheduled, HTTP, webhook) are disabled. You can still edit the pipeline and run manual test executions.

Debug Mode

Debug mode captures real incoming events and stores them as test events. This is useful for creating representative test data from actual production traffic. Disable when done to avoid storing unnecessary data.

Test events let you run your pipeline with sample data without affecting production. Create test events manually or capture them from live traffic using debug mode.

Each node in a run is tracked individually, allowing you to pinpoint exactly where failures occurred and inspect the data at each step.

The pipeline editor connects via WebSocket to receive live updates during execution. Nodes light up as they execute, showing status, duration, and errors in real-time.