Sending data from Cribl
Already route your logs through Cribl Stream or Cribl Edge? Keep it. Add LogPulse as a destination next to what you send today, compare both and move over source by source.
POST https://api.logpulse.io/api/v1/logs, format JSON Array, the LogPulse API key as Bearer token and gzip on. A small Eval sets index (the kind of data), source (the product) and CIM field names on each event. You can keep your current destination running next to it.Overview
Cribl sends to LogPulse over HTTPS with its Webhook destination. Each event goes to the HTTP API as JSON, with an API key in the Authorization header. A small Eval function in Cribl gives every event the shape LogPulse expects. Cribl Stream and Cribl Edge both have this destination, so the steps are the same.
- Firewalls to Route and shape
- Windows and Linux to Route and shape
- Cloud and apps to Route and shape
- Route and shape to Current destination (optional)
- Route and shape to HTTP API
- HTTP API to Ingest buffer
- Ingest buffer to Workers
- Workers to ClickHouse
| Step | What you do | Where |
|---|---|---|
| 1. API key | Create a key that may send logs | Configure → Sources → HTTP API |
| 2. Shape events | Set index, source, sourcetype and CIM field names | Cribl pipeline |
| 3. Destination | Add a Webhook destination with gzip and retries | Cribl destinations |
| 4. Check | Compare counts and fields per sourcetype | LogPulse search |
Which route to pick
| Route | Cribl destination | Use it when |
|---|---|---|
| HTTP API (recommended) | Webhook | Always a good fit. You choose index, source, sourcetype and fields per event. |
| LogPulse agent, HEC input | Splunk HEC | You already send Splunk HEC from Cribl and want to change only the URL. |
| OTLP | OpenTelemetry | The data is OpenTelemetry logs from applications. |
| LogPulse agent, syslog or Kafka input | Syslog or Kafka | Devices that only speak syslog, or a Kafka topic you already run. |
1. Create an API key
In LogPulse go to Configure → Sources → HTTP API and click New Key. Name it after the Cribl worker group, for example cribl-prod. Copy the key right away: it is shown once. An API key can only send data; it cannot search.
Store the key in Cribl as a secret (Group settings → Security → Secrets, type text) so it is not visible in the destination settings.
2. Shape events for LogPulse
Event format
The HTTP API takes a JSON array of events. LogPulse keeps only the keys below and drops the rest, so Cribl fields such as _raw, _time and cribl_* must be mapped first.
| Key | Type | What goes in |
|---|---|---|
timestamp | string | Event time in ISO 8601 UTC, ending in Z. Left out: the time of arrival. |
event | string or object | The raw line (usually _raw), or an object when Cribl already parsed it |
index | string | The kind of data, such as firewall. Left out: main |
source | string | The product, such as paloalto or fortigate |
sourcetype | string | The format, used by parser rules and detections |
host | string | The device or host that wrote the event |
level | string | info, warn, error, … Left out: LogPulse detects it |
fields | object of strings | Fields Cribl already extracted; stored as parsed fields |
attributes | object of strings | Extra metadata, such as the Cribl worker or pipeline |
[
{
"timestamp": "2026-11-15T14:02:11.000Z",
"event": "1,2026/11/15 14:02:11,0123456789,TRAFFIC,end,...",
"index": "firewall",
"source": "paloalto",
"sourcetype": "pan:traffic",
"host": "pa-fw-01",
"fields": {
"src": "10.1.2.3",
"dest": "203.0.113.10",
"dest_port": "443",
"action": "allowed",
"user": "alice"
},
"attributes": { "cribl_worker_group": "prod" }
}
]Values in fields and attributes must be strings. A number or a nested object there makes LogPulse reject that event, so cast with String(). When event is an object, LogPulse stores it as JSON and turns its keys into searchable fields itself; numbers are fine there.
Index and source
In LogPulse an index is a kind of data and the source is the product. Detection rules look in the index that belongs to their data, so a firewall event in main is missed by the firewall rules. Always set index: the HTTP API puts events without one in main.
| Index | For example |
|---|---|
firewall | Palo Alto, FortiGate, Cisco ASA, Check Point, pfSense, Sophos |
network | Routers, switches, wifi, NAC, VPN, load balancers |
windows | Windows Event Log and Sysmon |
linux | journald, auditd, sshd, sudo |
web | nginx, Apache, IIS and proxies |
dns | DNS queries |
email | Mail gateways and mail logs |
iam | Entra ID, Okta and other sign-in data |
endpoint | EDR such as Defender, CrowdStrike, SentinelOne |
audit | Admin activity in cloud services |
kubernetes | Pod logs and Kubernetes audit logs |
application | Your own application logs |
There are also containers, virtualization, vulnerabilities and cicd. If you send to Splunk today, a Cribl lookup from Splunk sourcetype to LogPulse index and source is the easiest way to set both:
sourcetype,lp_index,lp_source
pan:traffic,firewall,paloalto
pan:threat,firewall,paloalto
fortigate_traffic,firewall,fortigate
cisco:asa,firewall,cisco_asa
XmlWinEventLog,windows,windows
linux_secure,linux,linux
access_combined,web,apache
azure:aad:signin,iam,entraidField names (CIM)
LogPulse uses the field names of the Splunk Common Information Model: src, dest, src_port, dest_port, user, action, process, process_path, signature_id, http_user_agent and so on. Fields that arrive under these names work at once in detection rules, dashboards and searches you bring from Splunk.
If a Cribl pack already parses the data into CIM names, send them in fields. Otherwise rename in the same Eval, for example src_ip to src. Send the raw line in event as well: you keep the original, and a parser rule in LogPulse can add fields later without changing Cribl.
Cribl pipeline
Create a pipeline, for example logpulse_shape, and attach it to the destination as its post-processing pipeline. That way it only changes what goes to LogPulse, not what goes elsewhere. Add a Lookup function first if you use the CSV above, then one Eval function:
| Eval field | Value expression |
|---|---|
timestamp | new Date(_time * 1000).toISOString() |
event | _raw |
index | lp_index || 'main' |
source | lp_source || source |
sourcetype | sourcetype |
host | host |
fields | { src: String(src_ip), dest: String(dest_ip), dest_port: String(dest_port), action: String(action) } |
attributes | { cribl_worker_group: 'prod' } |
In the same Eval, set Keep fields to timestamp event index source sourcetype host level fields attributes and Remove fields to *. Keep wins over remove, so only the LogPulse keys are sent. Leave a field out of fields when the source has no value for it, rather than sending the text undefined.
3. Add the Webhook destination
In Cribl go to Data → Destinations → Webhook and add a new one. Setting names can differ a little between Cribl versions.
| Setting | Value |
|---|---|
| Output ID | logpulse |
| URL | https://api.logpulse.io/api/v1/logs |
| Method | POST |
| Format | JSON Array |
| Authentication | Token (text secret): the API key from step 1. Cribl sends it as Authorization: Bearer … |
| Post-processing pipeline | logpulse_shape |
| Compress | gzip |
On a dedicated LogPulse stack or an on-premises installation, use the API address of your own stack instead of api.logpulse.io; the path stays /api/v1/logs.
Then add the destination to a route, or to a QuickConnect source, next to the destination you use today. Cribl clones the events, so both keep receiving everything.
Batching and compression
LogPulse handles a thousand events in one request far better than a thousand requests with one event. Let Cribl fill batches:
| Cribl setting | Value | Why |
|---|---|---|
| Max body size | 4096 KB (the default) or up to 8192 KB | A request may be at most 10 MB, counted before compression |
| Max events per request | 0 (no limit) or 5000 | Batches are limited by size instead |
| Flush period | 1–5 s | Fuller batches at low volume |
| Request concurrency | 5 (the default) | Raise slowly if the queue grows |
| Compress | gzip | Usually 5–10 times less traffic |
One API key may make 10,000 requests per minute. A single event may be at most 256 KB; a longer one is stored cut to 256 KB and the reply says how many were cut. Your daily volume is counted on the size before compression, so gzip saves traffic, not volume.
Retries and backpressure
LogPulse answers 429 when a key sends too many requests or the daily volume of your plan is used up, and 503 when it cannot buffer the data for a moment. Both come with a Retry-After header. In the destination under Retries, retry on 429 and 503 and switch on Honor Retry-After header. Do not retry 400 or 401: the same request fails again.
Set Backpressure behavior to Persistent Queue, so Cribl keeps events on disk while LogPulse cannot take them instead of blocking your sources or dropping data.
207 with the number in the X-LogPulse-Refused-Events header. Cribl counts a 2xx as delivered, so give security data the Critical priority: per API key, per index or per sourcetype.TLS
The HTTP API only accepts HTTPS, with a publicly trusted certificate. Leave Validate server certs on and use no client certificate. If your network inspects TLS, let the proxy trust chain into Cribl or exclude api.logpulse.io from inspection. Workers need outbound access to it on port 443.
4. Check the data
Events are searchable shortly after they arrive. Compare the counts with what Cribl sent, per sourcetype:
earliest=-1h index=* source=paloalto
| stats count by index, sourcetype, hostThen check that the fields you search on are there, for example with earliest=-1h index=firewall | stats count by src, dest, action. Events LogPulse rejected are listed with the reason in index=_internal sourcetype=ingest_errors. In Cribl, the destination status shows sent, dropped and queued events.
Let both destinations run for a while. When the counts match and your searches and detections work in LogPulse, remove the old destination from the route.
Other routes
Splunk HEC to an agent
If Cribl already sends to Splunk with the Splunk HEC destination, you can point it at a LogPulse agent instead. Add a Splunk HEC input to an agent group on Linux or Kubernetes (port 8088 by default) and set the URL in Cribl to http://<agent-host>:8088/services/collector/event. List the HEC tokens the input may accept; with an empty list it accepts every request.
Events arrive with the sourcetype agent_splunk_hec. The agent keeps Splunk's sourcetype, source and index as splunk_sourcetype, splunk_source and splunk_index, so detection rules that look for a LogPulse sourcetype or index do not find this data by themselves; the Webhook route gives you that control. The HEC input on the agent listens without TLS, so use it only inside your own network. The agent itself sends to LogPulse over HTTPS.
OpenTelemetry
For OpenTelemetry logs, the OpenTelemetry destination in Cribl can send OTLP over HTTP to https://api.logpulse.io/v1/logs with the header Authorization: Bearer <API key>, protobuf or JSON, gzip on. Set the resource attribute logpulse.index to choose the index; without it events go to main. OTLP events get the sourcetype otlp, so for security data the Webhook route above is the better fit. See OTLP.
Syslog and Kafka
An agent can also receive syslog from Cribl on a syslog input (UDP or TCP), or read a Kafka topic Cribl writes to with a Kafka input. Both keep less control over index and fields than the Webhook route, so use them when the HTTP API is not an option, for example when Cribl may not reach the internet and only the agent may.
Troubleshooting
| What you see | Cause | What to do |
|---|---|---|
| 401 in Cribl | Key missing, wrong or revoked | Check the secret and that Authentication is Token, not Basic |
| 400 VALIDATION_ERROR | No event in the batch was valid | Look in index=_internal sourcetype=ingest_errors for the reason |
| Fewer events than sent | Single events were rejected | Usually a number in fields or a timestamp without Z; cast to String() and use toISOString() |
| Everything in main | No index on the event | Set index in the Eval, or check the lookup matches the sourcetype |
| Fields missing | Keep fields removed them, or another name was used | Check the Eval in preview; use CIM names |
| 429 or 503, queue grows | Too many small requests or a busy platform | Raise Flush period and Max body size; keep the persistent queue on |
| 429 QUOTA_EXCEEDED | The daily volume of your plan is used up | Retry-After points at midnight UTC, when the count starts again. Cribl keeps the events in its queue until then; send less or raise your plan |
Frequently asked questions
- Can Cribl send data to LogPulse?
- Yes. Cribl Stream and Cribl Edge send to LogPulse with the Webhook destination over HTTPS to https://api.logpulse.io/api/v1/logs, as a JSON array with an API key in the Authorization header. No plugin or pack is needed.
- Which Cribl destination do I use for LogPulse?
- The Webhook destination with format JSON Array. If you already send Splunk HEC from Cribl, you can also point the Splunk HEC destination at a LogPulse agent with a Splunk HEC input. OpenTelemetry logs can go to the OTLP endpoint /v1/logs.
- Can I keep sending to Splunk while I try LogPulse?
- Yes. Add LogPulse as a second destination on the same route. Cribl sends a copy to both, so you can compare counts and fields before you switch over.
- Which field names does LogPulse expect?
- The Splunk Common Information Model names, such as src, dest, dest_port, user, action, process and signature_id. Send them in the fields object as strings; the raw line goes in event.
- How does LogPulse pick the index for Cribl data?
- From the index key on each event. An index is a kind of data, such as firewall, windows or dns, and the source is the product. Events without an index go to main, where most detection rules do not look.