Before logs can be searched or analysed, they have to get from the servers, containers and services that produce them to your log platform. That job belongs to a log collection agent — also called a log shipper. This guide explains what a log collection agent does, the main options (Vector, Fluent Bit, Fluentd, Filebeat, the OpenTelemetry Collector), and how to choose.
What is a log collection agent?
A log collection agent (or log shipper) is a small program that runs on or near your systems, reads the logs they produce, optionally parses and transforms them, and forwards them to a central destination. It is the first stage of the log pipeline — without it, logs stay scattered on individual machines where they are useless for investigation or monitoring.
What a log collection agent does
- Collect — tail files, read container stdout, scrape system journals, or receive over a port.
- Parse & transform — structure raw lines, add metadata (host, environment), and optionally redact sensitive data.
- Buffer — hold events on disk or in memory so nothing is lost if the destination is briefly unavailable.
- Forward — ship the events to the log platform reliably, with retries and back-pressure.
The main log shippers
| Agent | Built with | Best known for |
|---|---|---|
| Vector | Rust | High-performance, modern pipeline for logs and metrics; rich transforms. |
| Fluent Bit | C | Extremely lightweight and fast; the default in many Kubernetes distros. |
| Fluentd | Ruby | Large plugin ecosystem; long-established, JSON-centric. |
| Filebeat | Go | Lightweight file shipper from the Elastic Beats family. |
| OpenTelemetry Collector | Go | Vendor-neutral, multi-signal (logs, metrics, traces) — the universal telemetry pipeline. |
There is no single winner: Fluent Bit excels at lightweight node-level collection, the OpenTelemetry Collector is the vendor-neutral choice for unified telemetry, and Vector is a high-performance pipeline with powerful in-flight transforms. Many estates even combine them (e.g. Fluent Bit at the node, a collector centrally).
How to choose
- Resource footprint — node and edge collection rewards lightweight agents.
- Reliability under load — look at buffering, back-pressure, and delivery guarantees at your throughput.
- Transforms — how much parsing, enrichment and redaction you need in-flight.
- Open standards — OTLP support avoids lock-in and unifies logs, metrics and traces.
- Operability — how easily you can deploy, configure and upgrade the fleet at scale.
Collection is also a security boundary
Agents run broadly across your estate and forward data out, so treat them as a security concern: authenticated transport, scoped credentials per agent, and central control over what each agent ships and where.
How LogPulse handles log collection
The LogPulse agent uses Vector as its runtime, wrapped in a control plane for fleet management: one-command install, server groups, per-agent authentication, and an EU-hosted control plane. It also accepts OTLP and a REST API, so you can ship with the OpenTelemetry Collector or any agent you already run. See Agent Fleet for the full picture, structured logging for getting clean data at the source, and what is log management for where collection fits in the pipeline.