# LogPulse — Full Content

> This is the expanded version of [llms.txt](https://logpulse.io/llms.txt) with complete blog post content and detailed documentation.

LogPulse is an AI-powered log management platform and an affordable alternative to Splunk, Datadog, and Elastic. Built on ClickHouse for sub-200ms search across billions of log entries.

## Pricing

- **Free**: Free — 1 GB/day, 30-day retention, 2 team members, 100 AI credits/mo
- **Pro**: €59/mo — 10 GB/day, 60-day retention, 10 team members, 2,500 AI credits/mo
- **Business**: €199/mo — 50 GB/day, 90-day retention, unlimited team members, 10,000 AI credits/mo
- **Business** includes Security Monitoring (SIEM) and Service Intelligence; lower tiers do not.
- **Enterprise**: Custom pricing — Fully dedicated stack (isolated ingest, compute, and storage), custom ingest limits, retention, SSO, and SLA

---

## Product Capabilities

## Service Intelligence (Observability)

Service Intelligence turns raw telemetry into a service-level view of the estate. URL: https://logpulse.io/docs/observability/service-intelligence.

- **Services**: A service is a logical thing you operate (a checkout API, an auth provider, a payments worker), assembled from the hosts, containers, and log sources that produce its telemetry. Membership is resolved automatically and stays current. A service is scoped either by **entity labels** (match every entity whose labels match a rule, e.g. team/app/env — new pods or hosts join on their own) or by **log source** (one or more source names, with no entity members).
- **KPIs**: A KPI (Key Performance Indicator) turns an LPQL search into a health signal. The query produces a value, a chosen value field is read as the metric, and a threshold maps it to a severity. A threshold has a **direction** ("above" flags high values like error rate or latency; "below" flags low values like success rate or throughput) and up to two breach levels — **warning** (early signal) and **critical** (objective breached). KPIs are evaluated on a schedule and can be charted over 1h / 6h / 24h / 7d.
- **Service health**: A service's status is rolled up from its KPIs — the most severe KPI wins. Statuses are Healthy (all KPIs within thresholds), Warning (a KPI crossed its warning level), Critical (a KPI crossed its critical level), and Unknown (no KPI has reported yet). The Observability hub summarizes the estate with Critical / Warning / Healthy counts and a status filter.
- **Dependencies**: Record upstream and downstream relationships between services and view them as a dependency graph, so a degraded service shows what it relies on and what relies on it.
- **KPI anomaly detection**: Each KPI is given a baseline — a per-KPI statistical profile of its normal range that accounts for daily and weekly seasonality — and deviations are flagged even while still inside the static thresholds. A per-KPI sensitivity controls how far from baseline a value must drift before it is flagged. Thresholds catch values you can name in advance; anomalies catch the ones you cannot.

## Security Monitoring (SIEM)

LogPulse Security Monitoring turns the existing log estate into a full SIEM on the same LPQL + ClickHouse engine. URL: https://logpulse.io/docs/security/monitoring. It is AI-evaluated and EU-sovereign: log data stays in the EU and AI evaluation runs on Amazon Bedrock in an EU region.

- **Detections & content packs**: 50+ built-in LPQL detections, each tagged with MITRE ATT&CK tactics and techniques, grouped into content packs (Authentication, Web, Network, DNS, Cloud/IAM, Endpoint, Email, Identity threats). Any LPQL query can be saved as a custom detection, and Sigma rules can be imported and transpiled to LPQL. Sequence (multi-stage) detections correlate ordered conditions for one entity within a time window to express attack chains.
- **Risk-Based Alerting & the effective risk score**: Instead of an alert per rule match, every signal emits a risk event attributed to an entity (user, host, IP). Every entity carries one **bounded 0–100 effective risk score** — the same number on the leaderboard, the notable, and the promotion decision. It uses diminishing returns (repeats of the same signal saturate instead of piling up, so volume does not outrank a real intrusion), an attack-breadth boost (activity spanning more MITRE tactics and techniques is weighted higher), time decay, and an entity-criticality multiplier. Promotion is one threshold on one score; there is one open notable per entity; a raised notable auto-resolves when risk decays below a clear threshold. Scores map to five bands: Info (0–19), Low (20–39), Moderate (40–69), High (70–89), Critical (90–100). The scoring is tunable per organization (threshold, decay half-life, volume sensitivity, tactic/technique diversity boosts, saturation, criticality multipliers).
- **Risk Events explorer, analyst dashboard & investigation**: A searchable stream of every risk event (filter by source engine, entity, MITRE tag, score, time), an analyst dashboard (top-risk-entity leaderboard by score and band, MITRE tactic breakdown, event-volume timeline, headline KPIs), and a full investigation workflow on each notable: live AI investigation status and verdict, an append-only notes trail, evidence pinning (snapshot contributing risk events so they survive retention), and manual escalation of a risk event into an analyst-owned notable that is exempt from auto-resolve. Risk events and notables are deep-linkable.
- **Threat intelligence**: A built-in IOC reputation store fed by free feeds — AlienVault OTX, AbuseIPDB, the abuse.ch suite (Feodo Tracker, URLhaus, ThreatFox), and the Spamhaus DROP (malicious network blocks via fast range match) and ASN-DROP (malicious autonomous systems via ASN resolution) blocklists. Matches boost risk and tag notables. Per-org allow/deny watchlists, plus the LPQL commands `| threatintel` (reputation lookup) and `| iplocation` (GeoIP).
- **UEBA, SOAR & coverage**: Behavioral analytics (impossible travel, first-seen activity, volume spikes) feed the same risk model. Notables are AI auto-investigated and benign ones auto-close. Response actions (notify, create ticket, block IP, disable user) run behind a human approval gate and can be composed into playbooks. A MITRE ATT&CK coverage heatmap is derived from enabled content, and compliance reports map NIS2, DORA, and ISO 27001 controls to the detections and data that evidence them.

## MCP Server (Agent Gateway)

LogPulse ships a remote Model Context Protocol (MCP) server so external AI agents can use the platform directly. Marketing page: https://logpulse.io/mcp. Documentation: https://logpulse.io/docs/integrations/mcp-server.

- **What it is**: A single HTTPS endpoint (`POST https://api.logpulse.io/mcp`, Streamable HTTP) that any MCP-capable client — Claude Code, Cursor, Codex, Continue, or a custom agent — connects to. Once connected, the agent discovers a catalog of typed LogPulse tools and calls them instead of scraping the dashboard or writing raw SQL. The tools are built on LPQL and run inside LogPulse, so answers are fast, consistent, governed, and tenant-scoped.
- **Authentication**: A LogPulse Personal Access Token (`lpat_`) sent as `Authorization: Bearer …`, or OAuth 2.1 (PKCE + dynamic client registration + browser consent) for one-click interactive sign-in. Both resolve to the same identity: organization, user, and scopes.
- **Scopes**: `logs:read` (log search/investigation + control-plane reads + data model), `siem:read` (risk events, notables, detections, UEBA, MITRE coverage, IOC lookups), `services:read` (service health, anomalies, dependencies, incidents), `entities:read` (Entity 360, blast radius, risk timeline), and the propose scopes `siem:propose` / `services:propose` / `pipelines:propose` (queue a detection/alert/service/KPI/pipeline/trigger for human approval — created disabled/inactive, destructive SOAR actions need owner approval; agents never write config directly). A token only sees the tools its scopes allow, and the SIEM (`siem:read` / `siem:propose`, plus `entities:read`) and Service Intelligence (`services:read` / `services:propose`) tools additionally require the Business plan that includes those features — Free/Pro tokens see only log-search and control-plane tools.
- **Tool catalog**: search_logs, count_patterns, timeline_analysis, compare_timeranges, get_field_values, search_similar_historical, system_health, lpql_help and control-plane list_/get_ reads (logs:read); search_risk_events, get_risk_summary, list_notables, get_notable, list_detections, get_detection, get_ueba_baselines, mitre_coverage, lookup_ioc, get_siem_settings (siem:read); get_service_health, list_anomalies, list_dependencies, list_services, list_incidents (services:read); get_entity, get_blast_radius, get_entity_risk_timeline (entities:read); propose_detection, propose_alert_rule (siem:propose), propose_service, propose_kpi (services:propose), propose_pipeline, propose_automation_trigger (pipelines:propose). It also exposes MCP resources (LPQL cheat sheet, detection guide, data models, saved queries) and prompts (triage_notable, is_ip_malicious, onboard_connector, service_health_check).
- **Security**: Read-only by default; writes are an explicit, scope-gated allowlist that only creates disabled drafts. The token holder's namespace RBAC is applied to every query. Every call is rate-limited (120/min/token), written to the audit log, and emitted to an internal telemetry stream watched by built-in detections (abnormal call volume or an agent write raises a notable). Tool output is treated as data, not instructions, to guard against prompt injection through logs. Data stays EU-hosted and AI evaluation runs in the EU. MCP access is part of the LogPulse plan; pricing stays flat (per plan, not per query or per agent).
- **Why it matters**: Engineers investigate logs, triage security, and build connectors from the agent they already use, without leaving their editor or moving data out of LogPulse. For AI-assisted development it closes the loop: after a deploy, the coding agent checks production logs for new errors itself, reads the real stack traces, writes the fix and verifies in the logs that the errors are gone, all in the same session.

---

## Guides

In-depth explainers on SIEM, log management, EU compliance (NIS2, DORA, ISO 27001, GDPR) and the agentic SOC. Each guide page carries Article + FAQPage structured data.

### NIS2 logging requirements explained

*Compliance · 9 min read · URL: https://logpulse.io/guides/nis2-logging-requirements*

What to log, how long to keep it, the monitoring duty, and the 24h/72h/1-month incident reporting deadlines under the NIS2 Directive.

**FAQ**

**How long do you need to keep logs under NIS2?**
NIS2 sets no single statutory retention period. It requires retention appropriate to the risk: long enough to detect, investigate and report an incident. In practice organisations keep logs for roughly 6 to 18 months, often split into a hot tier and an archive, and some national transpositions set their own minimums. Always check your national implementing law.

**Does NIS2 require a SIEM?**
NIS2 does not name any product category, but it requires continuous monitoring, correlation of events across sources, threshold-based alerting and a timely qualified response. That is the functional definition of a SIEM, so most in-scope organisations need SIEM-like capability to comply.

**What logs does NIS2 require you to collect?**
A defensible baseline covers authentication and access events, administrative and privileged actions, configuration and change events, network and security telemetry, application and system events, and access to sensitive data, each with accurate timestamps, identity, source and outcome.

**What are the NIS2 incident reporting deadlines?**
Under Article 23: an early warning within 24 hours, an incident notification within 72 hours, and a final report within one month. Your logs are the evidence base for each stage, so fast search and complete retention are essential.

**When did NIS2 take effect?**
NIS2 (Directive (EU) 2022/2555) had a transposition deadline of 17 October 2024. It applies through each Member State’s national law, and several countries transposed it late, so the binding details depend on your national legislation.

### NIS2 in Nederland: de Cyberbeveiligingswet uitgelegd

*Compliance · 10 min read · URL: https://logpulse.nl/guides/nis2-cyberbeveiligingswet*

Wie onder NIS2 valt, wat de zorgplicht en meldplicht inhouden, hoe het toezicht door de RDI werkt, en wat de Cyberbeveiligingswet concreet vraagt van je logging en monitoring.

**FAQ**

**Wat is de Cyberbeveiligingswet?**
De Cyberbeveiligingswet (Cbw) is de Nederlandse implementatie van de Europese NIS2-richtlijn en de opvolger van de Wet beveiliging netwerk- en informatiesystemen (Wbni). De wet is per 15 augustus 2026 van kracht, zonder overgangsperiode. Zij legt essentiële en belangrijke entiteiten een zorgplicht (beveiligingsmaatregelen, waaronder logging en monitoring), een meldplicht voor significante incidenten en een registratieplicht bij het NCSC op, met toezicht door de RDI en sectorale toezichthouders.

**Valt mijn organisatie onder NIS2?**
NIS2 geldt voor middelgrote en grote organisaties (vanaf circa 50 medewerkers of 10 miljoen euro jaaromzet) in aangewezen sectoren, van energie, transport, zorg en digitale infrastructuur tot post, afvalbeheer, chemie en digitale aanbieders. Sommige organisaties vallen er ongeacht omvang onder, bijvoorbeeld als enige aanbieder van een essentiële dienst. Doe de NIS2-zelfevaluatie van de Rijksoverheid om het formeel vast te stellen.

**Wat is het verschil tussen de Wbni en de Cyberbeveiligingswet?**
De Wbni implementeerde de eerste NIS-richtlijn en raakte enkele honderden aanbieders van essentiële diensten. De Cyberbeveiligingswet implementeert NIS2 en is veel breder: meer sectoren, ook middelgrote bedrijven, expliciete eisen aan risicobeheer (waaronder logging, detectie en incidentafhandeling), persoonlijke verantwoordelijkheid van bestuurders en fors hogere boetes.

**Hoe lang moet ik logs bewaren onder NIS2?**
NIS2 en de Cyberbeveiligingswet noemen geen vaste bewaartermijn. De norm is risicogebaseerd: lang genoeg om een incident te kunnen detecteren, onderzoeken en melden, inclusief de eindrapportage die binnen een maand af moet zijn. In de praktijk hanteren organisaties 6 tot 18 maanden, vaak gesplitst in een doorzoekbare hot-laag en een goedkoper archief.

**Maakt LogPulse mijn organisatie NIS2-compliant?**
Nee, geen enkel product maakt je vanzelf compliant. LogPulse dekt wel de logging-, detectie- en monitoringkant van de zorgplicht: gecentraliseerd logbeheer met instelbare retentie, realtime detecties met MITRE ATT&CK-mapping, incident-workflows die de 24-uurs- en 72-uursmeldingen onderbouwen, en compliance-rapportages die detecties aan NIS2-controls koppelen. De data blijft in de EU (GCP Amsterdam).

### De meldplicht onder de Cyberbeveiligingswet: het 24-uursdraaiboek

*Compliance · 8 min read · URL: https://logpulse.nl/guides/meldplicht-cyberbeveiligingswet*

Binnen 24 uur een vroegtijdige waarschuwing, binnen 72 uur de volledige melding, binnen een maand het eindrapport. Wat er in elke melding moet, bij wie je meldt, en de zes dingen die je vandaag klaarzet.

**FAQ**

**Wanneer moet ik een incident melden onder de Cyberbeveiligingswet?**
Bij een significant incident: een incident dat een ernstige operationele verstoring of financiële verliezen veroorzaakt of kan veroorzaken, of dat anderen aanzienlijke materiële of immateriële schade toebrengt of kan toebrengen. De meldketen kent drie deadlines: een vroegtijdige waarschuwing binnen 24 uur, een volledige melding binnen 72 uur en een eindrapport binnen één maand.

**Bij wie meld ik een incident?**
Bij het CSIRT (voor de meeste organisaties het NCSC) en bij de toezichthouder, voor veel sectoren de Rijksinspectie Digitale Infrastructuur (RDI). In de praktijk verloopt de melding via het meldportaal van het NCSC als één loket. Zorg dat je registratie en toegang tot dat portaal geregeld zijn vóórdat je ze nodig hebt.

**Wat als ik binnen 24 uur nog bijna niets weet?**
Dat is ingecalculeerd: de vroegtijdige waarschuwing mag beperkt zijn. Je meldt dat er een significant incident is, of je kwaadwillig handelen vermoedt en of er grensoverschrijdende gevolgen kunnen zijn. Wachten tot het volledige beeld er is, is de fout: de 72-uursmelding en het eindrapport zijn er voor de verdieping.

**Geldt de meldplicht ook voor belangrijke entiteiten?**
Ja. De meldplicht en de deadlines zijn identiek voor essentiële en belangrijke entiteiten. Het verschil zit in het toezicht (proactief versus reactief) en in de maximale boetes: tot 10 miljoen euro of 2% van de wereldwijde jaaromzet voor essentiële entiteiten, tot 7 miljoen euro of 1,4% voor belangrijke entiteiten.

**Hoe helpt logging bij de meldplicht?**
Elke fase van de meldketen leunt op logs: de 24-uursmelding op detectie die het incident opmerkt, de 72-uursmelding op doorzoekbare logs met indicatoren van compromittatie, het eindrapport op een volledige reconstructie met oorzaak. Gecentraliseerde logging met detectie, zoals LogPulse biedt, is de technische fundering; compliant word je pas met het bijbehorende proces en de mensen die het uitvoeren.

### NIS2 in Deutschland: das Umsetzungsgesetz erklärt

*Compliance · 10 min read · URL: https://logpulse.de/guides/nis2-umsetzungsgesetz*

Wer unter NIS2 fällt, was das NIS2-Umsetzungsgesetz an Risikomanagement- und Meldepflichten bringt, wie die BSI-Aufsicht funktioniert, und was das konkret für Logging und Monitoring bedeutet.

**FAQ**

**Was ist das NIS2-Umsetzungsgesetz (NIS2UmsuCG)?**
Das NIS2-Umsetzungs- und Cybersicherheitsstärkungsgesetz (NIS2UmsuCG) setzt die europäische NIS2-Richtlinie in deutsches Recht um, im Kern durch eine umfassende Novelle des BSI-Gesetzes (BSIG). Es ist seit dem 6. Dezember 2025 in Kraft, ohne Übergangsfrist für die Maßnahmen, und führt die Kategorien "besonders wichtige Einrichtungen" und "wichtige Einrichtungen" ein, mit Risikomanagement-Pflichten, einer dreistufigen Meldepflicht, einer Registrierungspflicht beim BSI und persönlichen Pflichten der Geschäftsleitung.

**Ist mein Unternehmen von NIS2 betroffen?**
NIS2 gilt für mittlere und große Unternehmen (ab etwa 50 Beschäftigten oder 10 Millionen Euro Jahresumsatz) in den regulierten Sektoren, von Energie, Transport, Gesundheit und digitaler Infrastruktur bis Post, Abfall, Chemie, verarbeitendem Gewerbe und digitalen Diensten. Insgesamt betrifft das in Deutschland Schätzungen zufolge rund 30.000 Unternehmen, ein Vielfaches der bisherigen KRITIS-Regulierung. Das BSI bietet eine offizielle Online-Betroffenheitsprüfung an.

**Was ist der Unterschied zwischen KRITIS und NIS2?**
Die KRITIS-Regulierung erfasst Betreiber kritischer Anlagen ab bestimmten Schwellenwerten und verlangt von ihnen bereits heute unter anderem Systeme zur Angriffserkennung. NIS2 ist deutlich breiter: Sie erfasst ganze Sektoren nach Unternehmensgröße statt nur einzelne Anlagen. KRITIS-Betreiber bleiben eine eigene, strenger regulierte Kategorie innerhalb des neuen Rahmens.

**Wie lange müssen Logs unter NIS2 aufbewahrt werden?**
Weder die Richtlinie noch das deutsche Umsetzungsgesetz nennen eine feste Aufbewahrungsfrist. Der Maßstab ist risikobasiert: lang genug, um Vorfälle zu erkennen, zu untersuchen und zu melden, einschließlich des Abschlussberichts nach einem Monat. In der Praxis üblich sind 6 bis 18 Monate, oft aufgeteilt in eine durchsuchbare Hot-Ebene und ein günstigeres Archiv.

**Macht LogPulse mein Unternehmen NIS2-konform?**
Nein, kein Produkt macht Sie von allein konform. LogPulse deckt aber die Logging-, Detektions- und Monitoring-Seite der Pflichten ab: zentralisiertes Log-Management mit konfigurierbarer Aufbewahrung, Echtzeit-Detektionen mit MITRE-ATT&CK-Zuordnung, Incident-Workflows, die die 24- und 72-Stunden-Meldungen mit Belegen unterfüttern, und Compliance-Berichte, die Detektionen NIS2-Anforderungen zuordnen. Alle Daten bleiben in der EU (GCP Amsterdam).

### NIS2 in the Netherlands: the Cyberbeveiligingswet explained

*Compliance · 10 min read · URL: https://logpulse.io/guides/nis2-netherlands-cyberbeveiligingswet*

The Netherlands implements NIS2 through the Cyberbeveiligingswet, in force from 15 August 2026 with no transition period. If you operate in the Netherlands but do not read Dutch, this guide explains who is covered, what the law requires, and what it means for your logging and monitoring.

**FAQ**

**What is the Cyberbeveiligingswet (Cbw)?**
The Cyberbeveiligingswet is the Dutch Cybersecurity Act that implements the EU NIS2 directive in the Netherlands, replacing the earlier Wbni. It enters into force on 15 August 2026 with no transition period: the duty of care, the incident notification duty and the registration duty with the NCSC apply immediately from that date.

**Does the Cbw apply to international companies with a Dutch entity?**
Yes, if the Dutch entity operates in one of the eighteen covered sectors and meets the size threshold of roughly 50 employees or 10 million euro in annual turnover. Smaller organisations can also be covered, for instance as part of a larger group or as the sole provider of an essential service. The Rijksoverheid offers an online NIS2 self-assessment, available in English, to check formally.

**What are the incident reporting deadlines under the Cbw?**
Significant incidents must be reported to the NCSC and the supervisory authority in three steps: an early warning within 24 hours, a full incident notification with an initial impact assessment and available IOCs within 72 hours, and a final report within one month. Meeting the 24-hour deadline requires that your logs already exist, are searchable, and that detection noticed the incident.

**Are directors personally responsible under the Cbw?**
Yes. The management board must approve the security measures, oversee their implementation and complete cybersecurity training. In cases of serious negligence, directors can be held personally liable, and for essential entities the regulator can, as a last resort, have directors temporarily suspended from their duties.

**How long must logs be retained under NIS2 in the Netherlands?**
The law sets no fixed retention period; the criterion is that you can investigate and report incidents. In practice, a risk-based 6 to 18 months is common, with a searchable hot tier and a cheaper archive. Document your choice and reasoning: that is exactly what the supervisory authority will ask for. No product makes you compliant by itself, but centralised logging with demonstrable retention is a foundation you can put in place today.

### NIS2 en France : où en est la loi et comment se préparer

*Compliance · 10 min read · URL: https://logpulse.fr/guides/nis2-france-loi-resilience*

La loi française de transposition de NIS2 n'est toujours pas adoptée, mais les exigences de la directive sont déjà connues : qui est concerné, ce que l'ANSSI attend, les délais de notification, et ce que cela signifie pour le logging et le monitoring.

**FAQ**

**La loi de transposition NIS2 est-elle adoptée en France ?**
Non. Le projet de loi relatif à la résilience des infrastructures critiques et au renforcement de la cybersécurité, qui regroupe trois volets (REC, NIS2 et DORA), a été adopté par le Sénat en première lecture le 12 mars 2025, puis dans une version amendée par la commission spéciale de l'Assemblée nationale le 10 septembre 2025. À ce jour, la loi n'est pas définitivement adoptée ni promulguée, et le 8 juillet 2026 la Commission européenne a saisi la Cour de justice de l'UE contre la France pour non-transposition, en demandant des sanctions financières.

**Qui sera concerné par NIS2 en France ?**
Environ 15 000 entités françaises devraient entrer dans le champ : les organisations moyennes et grandes (à partir d'environ 50 salariés ou 10 millions d'euros de chiffre d'affaires) dans les secteurs régulés, de l'énergie, des transports, de la santé et de l'infrastructure numérique jusqu'aux services postaux, aux déchets, à la chimie, à l'industrie manufacturière et aux fournisseurs numériques. L'ANSSI sera l'autorité compétente ; le secteur financier relève de l'ACPR et de l'AMF via DORA.

**Quels sont les délais de notification d'un incident sous NIS2 ?**
La directive fixe une chaîne en trois étapes : une alerte précoce sous 24 heures avec une première appréciation d'un éventuel acte malveillant, une notification complète sous 72 heures avec l'évaluation de la gravité et les indicateurs de compromission disponibles, et un rapport final sous un mois. Tenir ces délais suppose que les logs pertinents existent déjà, soient interrogeables et que la détection ait remarqué l'incident.

**Combien de temps faut-il conserver les logs pour NIS2 ?**
La directive ne fixe aucune durée de rétention : le critère est risk-based, c'est-à-dire assez longtemps pour détecter un incident, enquêter et le notifier, rapport final compris. En pratique, 6 à 18 mois sont courants, souvent répartis entre une couche interrogeable à chaud et une archive moins coûteuse. Ce choix et sa justification doivent être documentés.

**Faut-il attendre la loi française pour se préparer ?**
Non, c'est le piège. Les exigences sont déjà fixées par la directive (UE) 2022/2555, et l'ANSSI a annoncé qu'après la publication des référentiels techniques les entités disposeront d'environ trois ans pour se mettre pleinement en conformité, avec l'analyse des risques et les plans de sécurisation attendus dès fin 2026, puis le déploiement de mesures comme le MFA et les sauvegardes en 2027. Cartographier son SI et bâtir une vraie capacité de logging et de détection prend du temps : commencer maintenant, c'est transformer le retard législatif en avance opérationnelle.

### NIS2 en España: la transposición, las obligaciones y qué hacer ya

*Compliance · 10 min read · URL: https://logpulse.es/guides/nis2-espana-ley-ciberseguridad*

La ley española de transposición de NIS2 sigue en tramitación, pero los requisitos ya están fijados en la directiva. A quién afectará, qué exigirá y cómo preparar hoy tu logging y monitorización.

**FAQ**

**¿Ha transpuesto España ya la directiva NIS2?**
No. España incumplió el plazo del 17 de octubre de 2024. La transposición avanza mediante el Anteproyecto de Ley de Coordinación y Gobernanza de la Ciberseguridad, aprobado por el Consejo de Ministros el 14 de enero de 2025, pero a mediados de 2026 sigue en tramitación y no se ha publicado en el BOE.

**Si la ley española no está en vigor, ¿puedo esperar a que se publique?**
Esperar es una mala estrategia. Los requisitos esenciales ya están fijados en la Directiva (UE) 2022/2555: gestión de riesgos con logging y detección, notificación escalonada de incidentes y responsabilidad de la dirección. La Comisión Europea exigió en mayo de 2026 una transposición urgente, así que la ley puede llegar en cualquier momento y sin periodos de gracia generosos.

**¿Qué plazos de notificación de incidentes impone NIS2?**
Tres pasos: una alerta temprana en 24 horas, una notificación completa en 72 horas con evaluación de gravedad e indicadores de compromiso, y un informe final en un mes. Cumplirlos exige que los logs relevantes ya existan y sean consultables antes del incidente.

**¿Qué sanciones prevé el anteproyecto español?**
Multas de hasta 10 millones de euros para los incumplimientos más graves, además de facultades de supervisión proactiva sobre las entidades esenciales y reactiva sobre las importantes, y responsabilidad al nivel de la dirección por aprobar y supervisar las medidas.

**¿Cuánto tiempo debo conservar los logs para NIS2?**
La directiva no fija un plazo concreto: el criterio es poder investigar y notificar incidentes. En la práctica lo habitual es entre 6 y 18 meses, con una capa caliente consultable y un archivo más económico, documentando la decisión y su justificación.

### NIS2 in Italia: il D.Lgs. 138/2024 spiegato

*Compliance · 10 min read · URL: https://logpulse.it/guides/nis2-italia-dlgs-138*

Chi rientra in NIS2, cosa richiede il D.Lgs. 138/2024 in termini di gestione del rischio e notifica degli incidenti, come funziona la vigilanza dell'ACN, e cosa significa in concreto per logging e monitoraggio.

**FAQ**

**Cos'è il Decreto Legislativo 138/2024?**
Il Decreto Legislativo 4 settembre 2024, n. 138 recepisce la direttiva europea NIS2 nell'ordinamento italiano. È stato pubblicato in Gazzetta Ufficiale il 1° ottobre 2024 ed è in vigore dal 18 ottobre 2024. Introduce le categorie di soggetti essenziali e importanti, obblighi di gestione del rischio, una notifica degli incidenti in tre fasi, un obbligo di registrazione presso l'ACN e la responsabilità degli organi di amministrazione. Gli obblighi organizzativi di base sono scaduti il 18 aprile 2026.

**La mia organizzazione rientra in NIS2?**
NIS2 si applica alle medie e grandi organizzazioni (da circa 50 dipendenti o 10 milioni di euro di fatturato annuo) nei settori regolati, da energia, trasporti, sanità e infrastrutture digitali fino a servizi postali, rifiuti, chimica, manifatturiero e fornitori di servizi digitali. Alcune organizzazioni rientrano indipendentemente dalle dimensioni, ad esempio come unico fornitore di un servizio essenziale. Le entità nel perimetro devono registrarsi sul portale dell'ACN; chi entra nel perimetro in seguito si registra nella finestra annuale.

**Quali sono le scadenze di notifica degli incidenti sotto NIS2 in Italia?**
Gli incidenti significativi vanno notificati attraverso il CSIRT Italia, coordinato dall'ACN, in tre fasi conformi alla direttiva: un pre-allarme entro 24 ore, una notifica completa entro 72 ore con una prima valutazione di gravità, impatto e indicatori di compromissione, e una relazione finale entro un mese con causa probabile e misure adottate. Rispettare questi termini presuppone che i log esistano già, siano ricercabili e che il rilevamento abbia notato l'incidente.

**Per quanto tempo devo conservare i log sotto NIS2?**
Né la direttiva né il decreto italiano fissano un termine di conservazione. Il criterio è basato sul rischio: abbastanza a lungo da poter rilevare, indagare e notificare gli incidenti, inclusa la relazione finale dovuta entro un mese. Nella pratica le organizzazioni adottano 6-18 mesi, spesso divisi tra un livello "hot" ricercabile e un archivio più economico. Documenta la scelta e la motivazione: è ciò che l'autorità verifica in un audit.

**LogPulse rende la mia organizzazione conforme a NIS2?**
No, nessun prodotto rende conforme da solo. LogPulse copre però il lato logging, rilevamento e monitoraggio degli obblighi: gestione centralizzata dei log con retention configurabile, rilevamenti in tempo reale con mapping MITRE ATT&CK, workflow di incidente che sostengono le notifiche entro 24 e 72 ore, e report di compliance che collegano rilevamenti e dati agli obblighi NIS2. Tutti i dati restano nell'UE (GCP Amsterdam).

### DORA logging & monitoring requirements

*Compliance · 9 min read · URL: https://logpulse.io/guides/dora-logging-monitoring-requirements*

Detection under Article 10, what to log, retention, and the 4h/72h/1-month major-incident reporting clock for financial entities.

**FAQ**

**How long must financial entities keep logs under DORA?**
DORA sets no single statutory retention period. The ICT risk-management technical standards require you to define the retention period in a logging policy, justified by the events logged and your need to investigate and report incidents. In practice firms keep logs for a multi-month-to-multi-year window, often split into a hot tier and an archive, and frequently align with overlapping regimes such as the GDPR and PCI DSS.

**What are the DORA incident reporting deadlines?**
For a major ICT-related incident: an initial notification within 4 hours of classifying it as major (and no later than 24 hours after detection), an intermediate report within 72 hours of the initial notification, and a final report within one month of the latest intermediate report. The 4-hour clock starts at classification, not detection.

**Does DORA require a SIEM?**
DORA does not name any product category, but Article 10 requires continuous monitoring of user activity, anomalies and incidents, multiple layers of control, defined alert thresholds, and automatic alerting that initiates incident response. That is the functional definition of a SIEM with anomaly detection, so most in-scope firms need SIEM-like capability.

**When did DORA come into effect?**
DORA (Regulation (EU) 2022/2554) has been in application since 17 January 2025. As an EU regulation it applies directly across all Member States, without national transposition.

**Who must comply with DORA?**
A broad range of financial entities (banks, payment and e-money institutions, investment firms, insurers, crypto-asset service providers, trading venues and more), plus critical ICT third-party providers serving them. ICT suppliers to financial entities typically inherit DORA obligations through contracts.

### Log retention requirements by regulation

*Compliance · 8 min read · URL: https://logpulse.io/guides/log-retention-requirements*

How long to keep logs under PCI DSS, HIPAA, SOX, GDPR, ISO 27001, NIS2 and DORA: side by side, plus how to set a policy.

**FAQ**

**How long should you keep logs?**
There is no universal answer. Take the strictest retention requirement that applies to you and keep logs at least that long, while not holding personal data longer than the GDPR allows. Most organisations use a tiered policy: a hot tier (recent logs, fast to search) and a cheaper archive tier for the long-tail compliance and forensic window.

**What is the minimum log retention for PCI DSS?**
PCI DSS v4.0 (Requirement 10) requires retaining audit logs for at least 12 months, with at least the most recent 3 months immediately available for analysis.

**Does the GDPR set a log retention period?**
No. The GDPR applies the storage-limitation principle: keep personal data only as long as needed for the stated purpose, and justify the period. For security logs containing personal data, organisations commonly retain 1–3 years and redact where possible.

**What is the difference between hot and cold (archive) log retention?**
The hot tier holds recent logs kept fast to query for live monitoring, alerting and active investigation. The archive (cold) tier holds older logs in cheaper storage to satisfy long compliance windows and reconstruct incidents that surface months later. Splitting the two controls cost while meeting retention.

**How do you choose a log retention period?**
List every regulation, contract and sector rule that applies, take the strictest binding requirement, add a risk-based buffer for incident detection and forensics, decide your hot/archive split, apply GDPR storage-limitation to personal data, and document the period and its legal/risk basis.

### ISO 27001 logging requirements

*Compliance · 7 min read · URL: https://logpulse.io/guides/iso-27001-logging-requirements*

Annex A controls A.8.15 (Logging), A.8.16 (Monitoring) and A.8.17 (Clock sync): what to log, retention, and what auditors expect.

**FAQ**

**What does ISO 27001 require for logging?**
ISO 27001:2022 covers logging through Annex A controls A.8.15 (Logging: produce, store, protect and analyse logs of relevant events), A.8.16 (Monitoring activities: watch systems for anomalous behaviour), and A.8.17 (Clock synchronisation: sync clocks to an approved time source so timestamps are accurate and correlatable).

**How long do you need to keep logs for ISO 27001?**
ISO 27001 sets no fixed retention period. You determine it from your risk assessment, legal and contractual obligations, and incident-investigation needs, and document it in your ISMS. A commonly used baseline is at least 12 months; where other regimes apply, take the strictest.

**What is ISO 27001 control A.8.17?**
A.8.17 (Clock synchronisation) requires the clocks of all relevant information-processing systems to be synchronised to a single approved time source, typically via NTP, so log timestamps are accurate. Accurate time is essential for correlating events across systems during an investigation.

**What do auditors look for in ISO 27001 logging?**
Evidence that relevant events are logged centrally and protected from tampering, that logs are actively monitored and reviewed (not just stored), that clocks are synchronised (NTP config and drift monitoring), and that a risk-justified retention period is documented in the ISMS.

### GDPR & PII in logs

*Compliance · 7 min read · URL: https://logpulse.io/guides/gdpr-pii-in-logs*

Why logs contain personal data, the GDPR principles that apply, the right to erasure, and how to redact PII responsibly.

**FAQ**

**Does the GDPR apply to logs?**
Yes. Logs routinely contain personal data: usernames, user IDs, email addresses, and IP addresses (which EU case law treats as personal data in many contexts). A log store is therefore a store of personal data, and the GDPR’s principles apply to it.

**How long can you keep logs under the GDPR?**
The GDPR sets no fixed number but applies the storage-limitation principle: keep personal data only as long as needed for the stated purpose, and justify the period. For security logs, network and information security is a recognised legitimate interest (Recital 49), which commonly supports a defined retention window.

**How do you handle the right to erasure for logs?**
Security logging under a legitimate-interest or legal-obligation basis can justify retaining logs for a defined period despite an erasure request. Pseudonymising or redacting personal data in logs reduces the footprint so erasure is less often required in the first place.

**How do you handle PII in logs?**
Avoid logging secrets and unnecessary personal data at the source, redact or pseudonymise PII at ingest before storage, set and document a justified retention period, secure the store with access control and encryption, and, where EU law applies, keep log data in the EU.

### What is SIEM?

*Security · 9 min read · URL: https://logpulse.io/guides/what-is-siem*

What Security Information and Event Management is, what it does, how it works, and how the modern AI-era SIEM has evolved.

**FAQ**

**What does SIEM stand for?**
SIEM stands for Security Information and Event Management. It combines two earlier technologies: Security Information Management (SIM), for log storage and reporting, and Security Event Management (SEM), for real-time monitoring and alerting. The term was coined by Gartner analysts in 2005.

**What is the difference between SIM and SEM?**
SIM (Security Information Management) focuses on long-term storage, analysis and reporting of log data, the compliance and forensics side. SEM (Security Event Management) focuses on real-time monitoring and alerting, the live detection side. A SIEM is the combination of both on one platform.

**What does a SIEM do?**
A SIEM collects and normalises logs and security telemetry from across your environment, correlates events to detect threats, raises prioritised alerts, gives analysts fast search and context to investigate and respond (TDIR), and retains the data for forensics and compliance reporting.

**What is the difference between SIEM and log management?**
Log management focuses on collecting, storing and searching logs. A SIEM adds the security layer on top: correlation, threat detection, risk-based alerting, behavioral analytics (UEBA), and incident response, so it is purpose-built for finding and responding to attacks, not just storing logs.

**Do I need a SIEM for compliance?**
Frameworks like NIS2 and DORA rarely name SIEM outright, but they require continuous monitoring, correlation, alerting, retention and incident reporting, requirements that are very hard to meet at scale without SIEM-like capability.

### What is risk-based alerting (RBA)?

*Security · 7 min read · URL: https://logpulse.io/guides/what-is-risk-based-alerting*

How attributing risk to entities, instead of one alert per rule, cuts alert fatigue and surfaces the attacks that matter.

**FAQ**

**What is risk-based alerting?**
Risk-based alerting (RBA) is a SIEM approach where detections do not alert directly. Each detection emits a risk event attributed to an entity (user, host, IP) with a score; risk accumulates on the entity, and only when it crosses a threshold is a single notable case raised, carrying all the contributing context.

**How does risk-based alerting reduce alert fatigue?**
Because cases are raised per entity at a threshold rather than per rule match, RBA collapses thousands of low-fidelity alerts into a handful of high-confidence notables. Industry implementations report cutting low-fidelity alert volume by roughly 50–90%.

**What is a risk object and a risk score?**
A risk object is the entity that risk is attributed to, typically a user, host, or IP. The risk score is the accumulated risk on that entity. A good score uses diminishing returns so repeats saturate, an attack-breadth boost so activity across more MITRE stages weighs more, and time decay so old noise fades.

**How is RBA different from traditional SIEM alerting?**
Traditional alerting fires one alert per rule match, so volume equals importance and analysts drown. RBA attributes risk to entities, accumulates context across signals and MITRE tactics, and promotes one notable per entity at a single threshold, rewarding attack breadth over repetitive noise.

### What is an agentic SOC?

*AI & Agents · 8 min read · URL: https://logpulse.io/guides/what-is-an-agentic-soc*

AI agents that plan and run investigations: how it differs from SOAR, and why human-in-the-loop governance is the part that matters.

**FAQ**

**What is an agentic SOC?**
An agentic SOC is a security-operations model where AI agents can perceive context, plan a multi-step investigation, make decisions, and take or recommend actions toward a goal under human-defined constraints, rather than only answering a prompt or scoring an alert.

**How is an agentic SOC different from SOAR?**
SOAR and classic automation follow a fixed, pre-written decision tree. Agentic systems plan dynamically: they adjust the investigation path based on what they find and reason across multiple data sources in sequence, producing a reasoned verdict with evidence rather than just an action when conditions match.

**Does an agentic SOC replace analysts?**
No. The 2026 consensus is augmentation, not replacement, often called human-on-the-loop: AI handles alert volume at machine speed while humans own judgment, business context, novel threats, and consequential actions. Analysts shift from triaging alerts to supervising agent-led investigations.

**What does "human in the loop" mean in an agentic SOC?**
It means agents can investigate and draft changes freely, but any action that modifies your environment is gated behind human approval. Transparency, citations and an auditable trail, is what makes that supervision possible, and it is especially important for regulated teams.

### MCP for security: connecting AI agents to your logs

*AI & Agents · 8 min read · URL: https://logpulse.io/guides/mcp-for-security*

What MCP is, the real risks (prompt injection, tool poisoning, over-broad scopes), and how to connect AI agents to your logs safely.

**FAQ**

**What is the Model Context Protocol (MCP)?**
MCP is an open protocol that standardises how AI agents connect to external tools and data. An agent connects to an MCP server, discovers a catalog of typed tools, and calls them to read data or take actions, instead of scraping a UI or hand-writing queries. It is becoming the common interface between agents like Claude Code, Cursor and Codex and the systems they work with.

**Is it safe to connect an AI agent to your logs over MCP?**
It can be, if you follow least-privilege and treat data as untrusted. Connect read-only by default, scope access per token, enforce existing RBAC, route through an audited gateway with rate limits, require human approval for any change, and treat tool output (including log lines) as data, not instructions.

**What are tool poisoning and prompt injection in MCP?**
Prompt injection is malicious instructions hidden in data the agent reads (including your own logs) that try to hijack its behaviour. Tool poisoning embeds malicious instructions in a tool’s metadata or description, manipulating the agent through a trusted interface. Both are mitigated by treating retrieved content as data and constraining what tools can do.

**How does LogPulse secure MCP access?**
LogPulse’s remote MCP server is read-only by default, authenticated by a scoped personal access token or OAuth 2.1, bound by the user’s namespace RBAC, rate-limited and fully audited. The only writes are opt-in propose scopes that queue disabled drafts for human approval, tool output is treated as data to guard against prompt injection, and data stays EU-hosted.

### Closing the loop: production logs in your AI IDE

*AI & Agents · 8 min read · URL: https://logpulse.io/guides/production-logs-in-your-ai-ide*

Why AI-assisted development needs production feedback, and how MCP lets the agent that wrote the code verify deploys and debug errors from the IDE.

**FAQ**

**Can Claude Code read my production logs?**
Yes. Connect Claude Code to a log platform that ships a remote MCP server, such as LogPulse, and it can call typed tools like search_logs and count_patterns directly. The agent queries production logs, error patterns and service health from inside the coding session, read-only by default and scoped to your token.

**How do I connect Cursor or Codex to my logs?**
The same way as any MCP client: add the log platform’s MCP endpoint to the agent’s MCP configuration and authenticate with one-click OAuth or a scoped personal access token. LogPulse exposes one HTTPS endpoint (https://api.logpulse.io/mcp) that works with Claude Code, Cursor, Codex and any other MCP-capable agent.

**What does "closing the loop" mean in AI-assisted development?**
It means the AI agent that writes the code also sees the code run. After a deploy it queries production logs for new errors, reads the real stack traces, writes the fix and verifies in the logs that the errors are gone. The feedback cycle that used to run through dashboards and bug reports happens inside the coding session, in minutes.

**Is it safe to give a coding agent access to production logs?**
It can be, with the right guardrails: read-only access by default, per-token scopes, the user’s existing RBAC enforced on every query, rate limits, a full audit trail, and human approval for anything the agent proposes to change. Log content should be treated as data rather than instructions, to mitigate prompt injection.

**How do I debug an app built with vibe coding?**
Give the agent that built the app access to its production logs. Add structured logging in the first session, ship the logs to a platform the agent can query over MCP, and ask the agent why the app is failing: it reads the actual errors, sees when they started and which deploy caused them, and fixes the code it already knows.

### De loop sluiten: productielogs in je AI-IDE

*AI & Agents · 8 min read · URL: https://logpulse.nl/guides/productielogs-in-je-ai-ide*

Waarom AI-ontwikkeling productiefeedback nodig heeft, en hoe MCP de agent die de code schreef deploys laat verifiëren en fouten laat debuggen vanuit de IDE.

**FAQ**

**Kan Claude Code mijn productielogs lezen?**
Ja. Verbind Claude Code met een logplatform dat een remote MCP-server heeft, zoals LogPulse, en het kan getypeerde tools als search_logs en count_patterns rechtstreeks aanroepen. De agent bevraagt productielogs, foutpatronen en service-health vanuit de codeersessie, standaard read-only en begrensd tot de scopes van je token.

**Hoe koppel ik Cursor of Codex aan mijn logs?**
Zoals elke MCP-client: voeg het MCP-endpoint van het logplatform toe aan de MCP-configuratie van de agent en authenticeer met one-click OAuth of een gescopet personal access token. LogPulse biedt één HTTPS-endpoint (https://api.logpulse.io/mcp) dat werkt met Claude Code, Cursor, Codex en elke andere MCP-agent.

**Wat betekent "de loop sluiten" bij AI-ondersteund ontwikkelen?**
Het betekent dat de AI-agent die de code schrijft de code ook ziet draaien. Na een deploy bevraagt hij de productielogs op nieuwe fouten, leest de echte stacktraces, schrijft de fix en verifieert in de logs dat de fouten weg zijn. De feedbackcyclus die vroeger via dashboards en bugmeldingen liep, gebeurt binnen de codeersessie, in minuten.

**Is het veilig om een coding-agent toegang te geven tot productielogs?**
Dat kan, met de juiste guardrails: standaard read-only toegang, scopes per token, de bestaande RBAC van de gebruiker afgedwongen op elke query, rate limits, een volledige audittrail, en menselijke goedkeuring voor alles wat de agent wil wijzigen. Behandel loginhoud als data en niet als instructies, om prompt injection af te vangen.

**Hoe debug ik een app die met vibe coding gebouwd is?**
Geef de agent die de app bouwde toegang tot de productielogs van die app. Voeg gestructureerde logging toe in de eerste sessie, stuur de logs naar een platform dat de agent via MCP kan bevragen, en vraag de agent waarom de app faalt: hij leest de echte fouten, ziet wanneer ze begonnen en welke deploy ze veroorzaakte, en fixt de code die hij al kent.

### What is UEBA?

*Security · 7 min read · URL: https://logpulse.io/guides/what-is-ueba*

How behavioral baselines for users and entities catch compromised accounts, insider threats and lateral movement that static rules miss.

**FAQ**

**What is UEBA?**
UEBA (User and Entity Behavior Analytics) is a security technology that uses statistical analysis and machine learning to build a behavioral baseline for users and entities, then detects anomalies that may indicate a threat, asking whether behaviour is normal for this user or system, rather than whether it matched a known-bad rule.

**What does the "entity" in UEBA mean?**
An entity is a non-user actor: a host, server, application, service account, or IP address. Modelling entities as well as users lets UEBA catch threats with no human at the keyboard, such as a compromised service account or a server beaconing to a command-and-control host.

**How does UEBA work?**
UEBA baselines each user’s and entity’s normal pattern (login times, locations, data volumes, peer-group behaviour), compares current activity against that baseline accounting for daily and weekly rhythms, scores how abnormal the behaviour is, and attributes that risk to the user or entity.

**How is UEBA different from a SIEM?**
A SIEM collects and correlates evidence and runs detection rules; UEBA adds the behavioral layer, turning activity into a risk signal based on what is normal for each entity. UEBA is most useful as part of a SIEM, where it feeds risk-based alerting alongside detections and threat intel.

### What is SOAR?

*Security · 7 min read · URL: https://logpulse.io/guides/what-is-soar*

Security Orchestration, Automation and Response: the three pillars, playbooks, SOAR vs SIEM, and where the human stays in control.

**FAQ**

**What does SOAR stand for?**
SOAR stands for Security Orchestration, Automation and Response. It helps security teams act on incidents by connecting tools (orchestration), automating repetitive response steps (automation), and managing the incident workflow end to end (response). Where a SIEM detects, SOAR acts.

**What is a SOAR playbook?**
A playbook is a defined sequence of response steps that runs when a trigger fires: for example enrich the alert, open a ticket, notify the on-call analyst, and optionally contain the threat. Playbooks make response fast, consistent, and well-documented.

**What is the difference between SIEM and SOAR?**
A SIEM detects: it collects, correlates and alerts on security data. SOAR responds: it orchestrates tools, automates response steps, and manages cases, usually acting on alerts from the SIEM. Modern SIEMs increasingly embed SOAR-style response rather than requiring a separate product.

**Is automated response safe?**
Automating low-risk, reversible steps (enrich, ticket, notify) is safe. Destructive actions (disabling a user, blocking traffic, isolating a host) can disrupt the business on a false positive, so a sound default is to automate enrichment and gate destructive actions behind human approval, especially when AI agents drive the response.

### What is detection-as-code?

*Security · 7 min read · URL: https://logpulse.io/guides/detection-as-code*

Managing detections like software (Git, peer review, CI/CD testing and Sigma) for higher-quality, auditable, reversible detection content.

**FAQ**

**What is detection-as-code?**
Detection-as-code (DaC) treats detection rules as software artifacts: rules live as files in a version-controlled Git repository, changes go through peer-reviewed pull requests, automated tests in CI/CD verify they fire correctly, and deployment happens through a pipeline rather than manually in a console.

**What is Sigma in detection engineering?**
Sigma is an open-source, vendor-agnostic signature format for SIEM detections. You write the detection logic once and convert it to the native query language of your platform (SPL, KQL, Elastic DSL, LPQL and others). Being text-based and portable makes Sigma a natural fit for detection-as-code.

**Why use detection-as-code?**
It improves rule quality through peer review, gives full change tracking via Git history, enables fast rollback of a problematic rule, lets you replicate a vetted detection set across environments, and turns detection engineering into a collaborative practice with an auditable trail.

**How does detection-as-code relate to a SIEM?**
Detection-as-code is the delivery workflow for the detections your SIEM runs. Instead of editing rules by hand in the SIEM console, you author, version, test and deploy them as code, then push them to the SIEM programmatically.

### What is log management?

*Log management · 7 min read · URL: https://logpulse.io/guides/what-is-log-management*

How collecting, parsing, storing, searching and retaining logs works, and how it relates to observability and SIEM.

**FAQ**

**What is log management?**
Log management is the end-to-end handling of log data: collecting logs from across your estate, parsing them into a usable structure, storing them efficiently, making them searchable, alerting on patterns, and retaining them for as long as operations and compliance require.

**What is the difference between log management and a SIEM?**
Log management focuses on collecting, storing, searching and retaining logs. A SIEM adds the security layer on top: correlation, threat detection, risk-based alerting, behavioral analytics and incident response, so it is purpose-built for finding and responding to attacks, not just storing logs.

**Is log management the same as observability?**
No. Observability is about understanding system health and performance, usually across logs, metrics and traces. Log management is specifically the handling of logs, and it is the foundation that both observability and SIEM build on.

**What is centralized log management?**
Centralized log management collects logs from many sources into one place so they can be searched and correlated together, instead of sitting in silos on individual systems. It is a prerequisite for effective troubleshooting, monitoring and security investigation.

### SIEM vs log management vs XDR

*Log management · 7 min read · URL: https://logpulse.io/guides/siem-vs-log-management-vs-xdr*

What each does, how they differ on data scope, retention and response, and which one you actually need.

**FAQ**

**What is the difference between SIEM and log management?**
Log management collects, stores, searches and retains logs for everyone (ops, dev, security). A SIEM adds a security layer (correlation, threat detection, risk-based alerting and incident response) and long retention for compliance and forensics.

**What is the difference between SIEM and XDR?**
A SIEM is log- and compliance-centric: broad visibility, correlation, long retention, and investigation across your whole IT estate. XDR is threat- and response-centric: it unifies endpoint, network and cloud telemetry for fast, often automated detection and response, usually with shorter retention.

**Do I need all three?**
Not necessarily. Most organisations need log management as the base and add SIEM capability for security and compliance. XDR is a choice for teams that want integrated, automated detection and response across endpoints and network. The categories increasingly converge onto one engine.

**Which is best for compliance?**
SIEM and log management, because frameworks like NIS2 and DORA require long retention, correlation, monitoring and incident reporting. XDR’s shorter telemetry retention makes it less suited to long-term compliance on its own.

### ClickHouse vs Elasticsearch for logs

*Log management · 8 min read · URL: https://logpulse.io/guides/clickhouse-vs-elasticsearch-logs*

Columnar vs inverted index: compression, query speed and cost for log analytics, and where Elasticsearch still wins.

**FAQ**

**Is ClickHouse better than Elasticsearch for logs?**
For log analytics (high-volume, time-series, filter-and-aggregate workloads) ClickHouse is usually better: a columnar engine that compresses logs 5–10× and runs aggregation queries far faster, at much lower cost. Elasticsearch remains strong for full-text relevance search.

**Why is ClickHouse so much cheaper than Elasticsearch for logs?**
Elasticsearch stores each document multiple times (inverted index, stored fields, doc-values) and achieves modest compression, so it typically uses an order of magnitude more disk than ClickHouse, which uses columnar layout with strong (ZSTD) compression. Less storage and lower CPU overhead mean far fewer nodes for the same data.

**Does ClickHouse do full-text search?**
ClickHouse handles log search and filtering very well and has text-search capabilities, but Elasticsearch’s inverted index is purpose-built for full-text relevance ranking. If your workload is document/relevance search rather than log analytics, Elasticsearch may fit better.

**Should I migrate from Elasticsearch to ClickHouse for logs?**
If log analytics cost and query speed at scale are your pain points, ClickHouse is the common destination. The trade-off is full-text relevance features; for most log workloads (filter, aggregate, trend) the columnar engine wins on cost and speed.

### LPQL vs SPL (Splunk query language)

*Log management · 7 min read · URL: https://logpulse.io/guides/lpql-vs-spl*

Both share a pipe-based model, so Splunk queries translate easily. The difference is the engine, command breadth and AI generation.

**FAQ**

**Is LPQL like SPL?**
Yes. LPQL uses the same pipe-based model as Splunk’s SPL: a search expression narrows events, then results are piped through commands like stats, eval, where, top, sort and head. The everyday syntax is nearly identical, which makes moving between them easy.

**Can I migrate Splunk SPL queries to LPQL?**
Most everyday SPL searches translate to LPQL with little or no change because the filter-pipe-aggregate skeleton is the same. The work is in the long tail of SPL-specific commands and macros with no direct equivalent. Natural-language search lowers that cost further by generating the LPQL for you.

**What is different between LPQL and SPL?**
LPQL compiles to optimised ClickHouse SQL for columnar speed, focuses on a curated set of around two dozen pipe commands (versus SPL’s vast command set), has first-class JSON-path navigation for structured logs, and can be generated from natural language by the AI Investigator.

**Does LPQL support stats and eval like SPL?**
Yes. LPQL includes stats, timechart, eval, where, table, top/rare, dedup, sort, head/tail, fields, rename and rex, plus a similar eval-function set (string, math, conditional, time and pattern functions).

### Structured logging best practices

*Log management · 7 min read · URL: https://logpulse.io/guides/structured-logging*

What it is, why JSON beats free-form text, and the practices (schema, levels, context, correlation IDs, PII) that make it pay off.

**FAQ**

**What is structured logging?**
Structured logging records each event as machine-readable data, typically JSON key-value pairs, instead of free-form text. The information is the same, but fields like user_id or src_ip can be filtered, grouped and aggregated directly, without fragile text parsing.

**What is the difference between structured and unstructured logging?**
Unstructured logs are free-form sentences: easy to write and read, but hard to parse and aggregate. Structured logs put data in named fields, so they are reliably queryable, alertable and cheaper to store and search in a columnar engine. The cost is a little discipline at write time.

**What format should structured logs use?**
JSON is the de facto standard, supported by most logging libraries and collectors. Standards like OpenTelemetry (OTLP) push structured, consistently-attributed telemetry as the default, a good direction to align with.

**What are structured logging best practices?**
Use a consistent field schema across services, log in JSON, use proper severity levels, add context fields (service, environment, version), include correlation/trace IDs, use accurate UTC timestamps, keep message strings stable with variable data in fields, and avoid or redact PII.

### What is MITRE ATT&CK?

*Security · 7 min read · URL: https://logpulse.io/guides/what-is-mitre-attack*

The knowledge base of adversary tactics and techniques, the Enterprise matrix, and how teams use it for detection and coverage.

**FAQ**

**What is MITRE ATT&CK?**
MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is a free, globally-used knowledge base of real-world adversary behaviour maintained by the non-profit MITRE. It catalogues how attackers behave rather than malware signatures, which makes it durable as specific tools change.

**What is the difference between a tactic and a technique?**
A tactic is the attacker’s goal, the "why" of an action, such as Initial Access or Exfiltration. A technique is the "how", the method used to achieve a tactic, such as Phishing or Valid Accounts (T1078). Sub-techniques are more specific variations of a technique.

**How many tactics are in the MITRE ATT&CK Enterprise matrix?**
The Enterprise matrix has 14 tactics that roughly follow an attack’s progression, from Reconnaissance and Initial Access through Persistence, Privilege Escalation and Lateral Movement to Command and Control, Exfiltration and Impact.

**How do security teams use MITRE ATT&CK?**
Teams tag detections with the techniques they catch, map their detection coverage to find gaps, describe threat actors by the techniques they use, plan red/blue team exercises, and prioritise: an attack spanning many tactics is more serious than one noisy technique repeated.

### What is an AI SOC analyst?

*AI & Agents · 7 min read · URL: https://logpulse.io/guides/ai-soc-analyst*

Software that triages alerts, investigates and writes verdicts at machine speed, what it does well, and where humans stay essential.

**FAQ**

**What is an AI SOC analyst?**
An AI SOC analyst (or AI SOC agent) is an AI system that performs front-line security-operations tasks: evaluating and enriching alerts, investigating, and producing a reasoned verdict with evidence. It is the analyst-shaped expression of the broader agentic SOC.

**Does an AI SOC analyst replace human analysts?**
No. The mature model is augmentation (human-on-the-loop): the AI handles alert volume and repetitive triage at machine speed, while humans own judgment, business context, novel threats, and consequential actions. Analysts shift from triaging alerts to supervising agent-led investigations.

**What does an AI SOC analyst do?**
It triages alerts (true vs false positive with context), investigates by gathering evidence and correlating signals, delivers a verdict with the reasoning, and recommends containment or response for a human to approve.

**How do you trust an AI SOC analyst?**
Through transparency: insist on citations, an evidence trail, and a human-approval gate before any action touches your environment. An AI analyst that shows its evidence can be verified and overridden; a black box cannot, which matters especially for regulated teams.

### AI root-cause analysis

*AI & Agents · 6 min read · URL: https://logpulse.io/guides/ai-root-cause-analysis*

How an AI agent queries logs, correlates deploys and changes, and proposes a likely cause with evidence. Verify in minutes, not hours.

**FAQ**

**What is AI root-cause analysis?**
AI root-cause analysis (RCA) is AI-assisted investigation of why an incident happened. Given a symptom (an error spike, a latency jump, a failing service) an AI agent searches the relevant logs and telemetry, correlates them with deploys and configuration changes, and proposes the most likely cause with supporting evidence.

**How does AI root-cause analysis work?**
It starts from a symptom or natural-language question, generates and runs queries, correlates the timeline with recent deploys and config changes, pivots on what it finds to follow the trail, and proposes a likely root cause with linked evidence for a human to verify.

**Does AI root-cause analysis replace engineers?**
No. It does the tedious hunting so the engineer verifies a conclusion instead of starting from a blank query box. The human stays the owner of the conclusion; transparency (showing the queries and evidence) is what makes that verification possible.

**How does LogPulse do AI root-cause analysis?**
The LogPulse AI Investigator turns a natural-language question into LPQL, runs it on ClickHouse, correlates the timeline with deploys and changes, and proposes a likely root cause with linked evidence, showing every query so you stay in control. It is included on every plan.

### What is a log collection agent?

*Log management · 7 min read · URL: https://logpulse.io/guides/what-is-a-log-collection-agent*

What log shippers do, how Vector, Fluent Bit, Fluentd, Filebeat and the OpenTelemetry Collector compare, and how to choose one.

**FAQ**

**What is a log collection agent?**
A log collection agent (or log shipper) is a small program that runs on or near your systems, reads the logs they produce, optionally parses and transforms them, buffers them, and forwards them to a central log platform. It is the first stage of the log pipeline.

**What are the main log shippers?**
Common options are Vector (Rust, high-performance pipeline), Fluent Bit (C, extremely lightweight, default in many Kubernetes distros), Fluentd (Ruby, large plugin ecosystem), Filebeat (Go, lightweight file shipper from Elastic Beats), and the OpenTelemetry Collector (vendor-neutral, multi-signal for logs, metrics and traces).

**Which log collection agent should I use?**
It depends on resource footprint, reliability under your throughput, how much in-flight transformation you need, open-standards (OTLP) support, and operability at scale. Fluent Bit suits lightweight node-level collection, the OpenTelemetry Collector suits vendor-neutral unified telemetry, and Vector is a strong high-performance pipeline. Many estates combine them.

**What agent does LogPulse use?**
The LogPulse agent uses Vector as its runtime, wrapped in a control plane for fleet management (one-command install, server groups, per-agent authentication, EU-hosted control plane). LogPulse also accepts OTLP and a REST API, so you can ship with the OpenTelemetry Collector or any agent you already run.

---

## Blog Posts

### NIS2 Logging Requirements: A Practical Checklist

*By Gianno Kardjo · June 28, 2026 · 7 min read · Guides*
*URL: https://logpulse.io/blog/nis2-logging-requirements-checklist*

NIS2 has spent two years as a directive on a slide. In 2026 it becomes something concrete: active supervision, inspections, and, in the Netherlands, the Cyberbeveiligingswet that transposes it into national law. The shift from "we should look at this" to "an auditor is asking" lands hardest on one team: whoever owns logging and monitoring.

The good news is that most of what NIS2 expects from your logs is not exotic. It is the discipline of collecting the right events, keeping them long enough, watching them in real time, and being able to prove all of it. This is a practical checklist of what that means day to day, and how to evidence each item. For the platform-level summary, see our [NIS2 compliance](/compliance/nis2) page.

#### What NIS2 actually asks of your logs

NIS2 does not hand you a list of log fields. It sets outcomes: you must be able to detect incidents, respond to them, and report significant ones quickly: an early warning within 24 hours and a fuller notification within 72. Everything in the logging domain exists to make those outcomes possible and provable.

Translated into operational terms, four capabilities have to hold together: you can see events from across your estate in one place, you keep them long enough to investigate and to satisfy an auditor, you spot the bad ones as they happen rather than weeks later, and you can produce evidence on demand. Miss any one of those and the others stop counting.

#### The checklist

Centralized collection. Logs from servers, network devices, identity providers, cloud services, and critical applications land in one queryable place. Scattered logs on individual hosts are not monitorable and are trivially tampered with.

Adequate, tamper-resistant retention. Events survive long enough to investigate an incident discovered months later and to evidence controls during an audit. The store should be append-only so a logged event cannot be quietly edited away.

Real-time detection mapped to known techniques. Raw collection is not monitoring. You need detections, ideally mapped to a framework like MITRE ATT&CK so coverage is legible, that raise something a human can act on as events arrive.

An incident workflow that can hit the clock. When something fires, you need to triage, confirm, and escalate fast enough to meet the 24- and 72-hour duties, with a record of what was decided and when.

Evidence you can hand to a regulator. Coverage reports, an audit trail of who did what, and the ability to reconstruct an incident timeline. "We had logs somewhere" is not evidence; a report that maps controls to data is.

Access control and an audit trail on the logs themselves. Who can read and change the logging configuration is itself in scope. The monitoring system has to monitor its own use.

#### Mapping the checklist to a platform

This is the shape LogPulse is built around, which is why the checklist maps cleanly. Collection is centralized into one EU-hosted store; retention is configurable per plan with an append-only audit log; detection ships as 50+ MITRE ATT&CK-tagged rules plus anything you write in LPQL.

On top of that sits the part that actually saves an analyst time during an incident: instead of an alert per rule, every signal contributes to one bounded risk score per entity, and a small number of high-confidence notables are raised, AI-investigated, and worked through an analyst dashboard with notes, evidence, and escalation. Compliance reports then map the detections and data to NIS2 controls. The [Security Monitoring (SIEM)](/siem) page walks through that engine in detail.

#### What a platform cannot do for you

It is worth being blunt about the boundary, because vendors usually are not. No tool makes you NIS2-compliant. A platform covers the logging, detection, monitoring, and reporting layer and produces evidence. It does not determine whether NIS2 applies to you, write your risk-management policy, run your governance, or train your people. Those are organizational, and they stay yours.

The honest framing is that a good platform removes the technical excuse. Once collection, detection, and reporting are handled, the remaining work is process and governance, which is where it should be, not stuck on "we cannot even see our logs."

#### Where to start

You do not need a compliance program to begin; you need visibility. Start by centralizing logs from your most critical systems and turning on the built-in detections, then iterate toward the rest of the checklist. The free plan is enough to get logs flowing and see the model work; Security Monitoring and the NIS2 control reports live on the Business plan.

If NIS2 is on your roadmap for 2026, the cheapest move you can make today is to stop logging into silos. Everything else on the checklist gets easier once the data is in one place. Start with the [NIS2 compliance](/compliance/nis2) overview, or just ship your first logs and build from there.


### How to Reduce SOC Alert Fatigue with Risk-Based Alerting

*By Gianno Kardjo · June 26, 2026 · 6 min read · Guides*
*URL: https://logpulse.io/blog/reduce-soc-alert-fatigue-risk-based-alerting*

Ask anyone who has worked a SOC shift what the job actually feels like, and the answer is rarely "hunting threats." It is closing tickets. The average security operations center processes thousands of alerts a day, a large fraction of them false positives, and the people doing it report being permanently behind. This is alert fatigue, and it is not a motivation problem. It is a design problem.

The instinct is to fix it with more tuning: suppress this rule, raise that threshold, add an exception. Tuning helps at the margin, but it never ends, because the underlying model guarantees the flood. The real fix is to change what raises an alert in the first place.

#### Why per-rule alerting fails

In a classic SIEM, every detection rule fires on its own. A brute-force rule, an impossible-travel rule, a new-admin rule: each evaluates independently and each emits its own alert. That feels thorough, but it has two fatal properties at scale.

First, volume is mistaken for severity. A single noisy rule matching benign automation can out-produce a genuine intrusion, so the real signal is buried under the loud one. Second, the alerts are not connected. Five weak signals on the same user (a first-seen login, a new device, an odd hour, a volume spike, a sensitive access) arrive as five separate tickets instead of one obviously suspicious entity. The analyst has to reassemble by hand what the system pulled apart.

#### Risk-based alerting: score the entity, not the rule

Risk-based alerting inverts the model. Instead of an alert per rule match, every signal emits a risk event attributed to an entity: a user, a host, an IP. Each entity then carries one bounded score from 0 to 100, and that single number is what decides whether a human gets involved.

The scoring is designed to resist exactly the failure modes above. Repeats of the same signal saturate through diminishing returns, so a noisy rule cannot pile up and outrank a real attack. Activity that spans more MITRE tactics and techniques is weighted higher, because breadth is a better intrusion signal than volume. Scores decay over time, and a per-entity criticality multiplier means the same behavior on a domain controller matters more than on a test box. Promotion becomes one threshold on one number, and there is one open notable per entity instead of a queue of fragments. The [Security Monitoring (SIEM)](/siem) engine is built on this model.

#### From alert to answer

Reducing the count is only half the win. The other half is what happens to the alerts that survive. When a notable is raised, it is auto-investigated by an LLM that triages it and closes the benign ones before a human ever opens them, then presents the rest as a workspace with the contributing evidence, notes, and escalation already in place.

This is where alert fatigue and the [agentic SOC](/agentic-soc) meet. The agent does the repetitive first pass (gathering context, ruling out the obvious false positives, drafting a verdict) and the analyst spends their attention on the handful of findings that genuinely need judgment. Fewer alerts, and the ones that remain arrive already half-investigated.

#### What changes for the analyst

The day-to-day shift is large. Instead of triaging thousands of independent alerts and manually correlating them, an analyst opens a short list of entities ranked by a score they can trust, each already enriched and investigated. The work moves from clearing a queue to making decisions, which is the work analysts are actually good at and the reason they took the job.

#### Getting started

Risk-based alerting is not a separate product to buy; it is how detection works in LogPulse Security Monitoring. The scoring is tunable per organization (the promotion threshold, decay half-life, volume sensitivity, and criticality multipliers are all knobs) so you can match it to your estate instead of fighting a fixed model. Start free to get logs flowing, then turn on Security Monitoring when you are ready to trade your alert queue for a leaderboard of the entities that actually matter.


### Agentic SOC vs Traditional SIEM: What Actually Changes

*By Gianno Kardjo · June 24, 2026 · 7 min read · Guides*
*URL: https://logpulse.io/blog/agentic-soc-vs-traditional-siem*

In the survey data for late 2025 and early 2026, something close to three-quarters of security teams said they were already using or building agentic AI. "Agentic SOC" became the loudest phrase in the industry almost overnight. Behind the noise there is a real shift, but it is narrower and more interesting than the marketing suggests.

The change is not "AI replaces the SOC." It is a change in who does the investigating, and what the human is for. This piece is about what actually moves when agents enter the SOC, and the guardrails that decide whether that is an upgrade or a liability.

#### What "agentic" actually means

Automation is not new to security. Playbooks have run scripted responses for years: if this alert, then that action. What makes an agent different is that it is not following a fixed script. It can reason about context, decide which question to ask next, pull the data it needs, and adapt its path as it learns, the way a junior analyst would, rather than the way a cron job does.

That is the whole distinction worth holding onto. A traditional automation does the same thing every time. An agent decides what to do. Everything good and everything dangerous about the agentic SOC follows from that one property.

#### What changes versus a traditional SIEM

A traditional SIEM is a pipeline: rules match events, alerts are emitted, and humans triage them. The intelligence lives in the rules someone wrote in advance, and the human is the first responder to every firing.

In an agentic model, the agent moves to the front of the investigation. It triages the incoming signal, gathers context across sources, forms a hypothesis, and drafts a verdict or a response, before a human is pulled in. The analyst is no longer the first responder to every alert; they become the reviewer of agent-led work and the decision-maker on the cases that matter. The role shifts from doing the investigation to supervising it.

#### The risk: handing over the keys

This is genuinely powerful, and it is also where it goes wrong if you are careless. An agent that can take actions on its own is an agent that can take the wrong action at machine speed. And security data is a hostile input: logs can contain attacker-controlled text, so an agent that treats tool output as instructions can be steered through prompt injection, told, in effect, by the very data it is investigating to do something it should not.

The failure mode of a naive agentic SOC is not that the AI is dumb. It is that it is trusted too much: given autonomy to change detections, disable users, or block traffic without a human in the path, and fed data that an adversary can write to.

#### Human-in-the-loop is the answer

The version that is safe to run is propose-and-approve. Agents can investigate freely and draft anything (a new detection, an alert rule, a response playbook), but what they draft is created disabled and applied only after a human approves it. Destructive actions require an extra owner sign-off. Nothing reaches production on the agent's say-so alone.

The other half is treating the agent like an untrusted client of the platform, not a privileged insider. In LogPulse, agents connect over the [MCP server](/mcp) with read-only access by default, scoped per token, rate-limited, and audited; tool output is handled as data, not as instructions, to blunt prompt injection; and abnormal agent behavior is itself watched by built-in detections. That is what makes "agentic" an upgrade instead of a new attack surface. The [agentic SOC](/agentic-soc) page goes through the model end to end.

#### Where to start

You do not adopt an agentic SOC by flipping a switch. The sensible path is to start with the foundation (a risk-based SIEM where notables are already AI-investigated and false positives close themselves) and then let agents act on that foundation through a governed gateway. The investigation gets faster without the control getting looser. Begin with [Security Monitoring (SIEM)](/siem), and add agents over MCP once the basics are paying off.


### Pipelines: The Pillar Your Log Analysis Stands On

*By Gianno Kardjo · May 7, 2026 · 6 min read · Product*
*URL: https://logpulse.io/blog/pipelines-the-pillar-your-log-analysis-stands-on*

Garbage in, garbage out is not a slogan in log analysis. It is the entire game. The dashboards you build, the alerts you trust, the AI investigations you run, the SIEM detections you write. All of them are downstream of one question: is the data that lands in your index actually useful, or is it raw noise dressed up as evidence?

We treat the pipeline mechanism as one of the load-bearing pillars of LogPulse, equal in weight to LPQL and to AI-assisted investigation. Not a side feature, not a power-user tool, not a bolt-on for after the index is already polluted. A first-class layer between every source and your storage, where you decide what to keep, what to drop, what to enrich, and what to mask before a single byte becomes queryable.

#### Shape Is Decided at Ingest, Not at Query Time

Walk into any mature log environment and ask what slows the operators down. The answer is rarely "the search is too slow." It is "I have to wade through twelve fields of nginx noise before I find the request_id," or "auth errors come in three different shapes depending on the version," or "half of these events contain personal data and we can't expose them to the wider team." These are not search problems. They are shape problems.

When the shape is wrong, every layer above it suffers. Dashboards become fragile because the field they depend on appears in 60% of events instead of 100%. Alerts misfire because `level` is sometimes a string and sometimes a number. RBAC leaks because legacy sources lack the namespace label. AI investigation produces weaker findings because the model reasons over inconsistent evidence. The fix is the same in every case: do the work earlier in the chain. Skip the pipeline layer and you spend the rest of the product's lifetime working around the data you accepted on day one.

#### What a Pipeline Actually Does

A LogPulse pipeline is a directed graph. Data enters at a trigger, flows through a chain of transformation nodes, and lands at a destination. Each node performs one scoped operation: parse a JSON body, mask an email, enrich an IP with a geo lookup, drop debug events nobody will ever query, split a batch payload into one event per row, coerce a custom `level` field into the canonical info/warn/error scale.

Splitting the work into nodes is not about visual elegance. It is about auditability. When something is wrong in production, you do not want to reverse-engineer 600 lines of Logstash conf. You open the pipeline, click the node where the field changed shape, and inspect that exact run's input and output. The engine in `packages/pipeline-engine` enforces a per-node timeout, scrubs secrets out of run logs before anything is written, and emits structured run events that you can query in LPQL like any other source. The pipeline is observable as data, not as a black box.

**A typical ingest-shaping pipeline**
```text
start
  → httpRequest        (pull raw events from a webhook or API)
  → json               (parse the body if delivered as a string)
  → split              (one upstream payload → one log event per row)
  → transform          (rename fields, drop noise, coerce types)
  → redactMask         (mask emails, card numbers, tokens)
  → mapCommonSchema    (normalise into LogPulse's canonical shape)
  → logpulseIngest     (write to the right index with the right metadata)
```

#### Transforming Data Until What Remains Is Valuable

The most underused word in log analysis is "drop". Engineers are trained to keep everything because storage is cheap and you might need it later. That instinct made sense when logs were a backup; it actively harms you when logs are the primary signal you reason over. Every event you keep that nobody will ever query is noise other queries have to scan past, a row the Investigator has to consider, a field the dashboard has to filter out. Keeping less is not a compromise. It is the work.

Pipelines are where that work happens reviewably. A condition node samples debug events down to one in a hundred. A transform strips the seventeen Kubernetes annotations nobody queries and keeps the three that matter. A redactMask ensures the customer email an over-eager logger emitted never becomes part of the indexed corpus. Enrichment is the same pattern in reverse: a lookup node joins inbound events against a reference dataset (a CIDR list of internal networks, a service-to-team mapping, a list of maintenance windows) and "request from 10.4.18.22 took 4200ms" arrives downstream tagged with the owning team, the environment, and a flag indicating it happened during a planned deploy. That is what turns a log into evidence.

#### The AI Pipeline Agent: Quality Without the Regex Archaeology

Here is the honest tension. Everything above is real engineering work. It assumes you know what your logs look like in the wild, what every integration's payload shape is, which regex actually matches your correlation IDs, and which fields contain personal data. For experienced platform engineers this is doable. For everyone else it is the wall pipelines have always run into: Logstash configs failed not because pipelines are a bad idea, but because the activation energy was too high.

The AI Pipeline Agent lowers that activation energy without lowering the quality bar. It is a chat agent on the same Claude-backed loop that powers our Investigator, pointed at the canvas you are looking at. You describe what you want in natural language. The agent reads the current graph, the most recent run output per node, and the engine's structured run logs in your tenant's indexes. It can call out to web search, gated by a one-click consent prompt, to confirm an integration's exact payload shape before building against it. Then it composes the pipeline node by node and shows you on the canvas what it added or modified, every touched node badged "added" or "modified" until you accept the draft.

The agent is good at the parts humans are bad at: knowing the shape of a third-party API on the first try, remembering which transform op takes a plural `targets` array versus a singular `target`, catching that you wired the destination index back into the source and created an ingestion loop. It will not silently take a destructive action (bulk-delete every node, wipe edges, drop a redaction step) without explicit confirmation. It defaults to extracting webhook URLs and credentials into pipeline secrets rather than inlining them, because the engine's secret-scrubber only redacts values it knows about. None of this is the agent being clever; it is the agent applying rules we know from operating these pipelines ourselves.

**A typical Pipeline Agent session**
```text
you: every hour, pull our staging error logs from the last hour,
     mask any customer emails, group by service, and POST a markdown
     digest to https://hooks.slack.com/services/...

agent: I'd like to confirm the Slack incoming-webhook request shape
       via web search before I build this. OK to look it up?

you: yes

agent: Slack expects POST { "text": "..." } and a 2xx body counts as
       delivered. Building this as five nodes:

         start → lpqlSearch → redactMask → transform → httpRequest

       - lpqlSearch runs `level=error earliest=-1h` against your tenant
         logs and lands the rows on the payload.
       - redactMask masks email-shaped strings in the rows.
       - transform groups by `service` and folds the rows into a
         markdown summary string.
       - httpRequest POSTs { text: "<summary>" } to your webhook.

       Trigger set to schedule every hour. Five nodes badged "added"
       on the canvas. Want me to run it once now to verify Slack
       delivery before we promote the draft?
```

#### Where to Start

If you are evaluating LogPulse, do not start with the search bar. Pick one source that matters (a Kubernetes cluster, a Datadog forwarder you are migrating off, an internal app that emits JSON, a webhook from a SaaS you depend on) and build the pipeline first. Open the canvas, talk to the agent, watch the badges appear, run it once, look at what landed in the index. The quality of every search, alert, and investigation after that point is decided in the five minutes you spend here.

A good pipeline mechanism is not the part of the platform that demos well. It is the part that, six months in, you stop noticing because everything downstream just works. That is the bar we hold ourselves to, and the reason pipelines sit alongside LPQL and the Investigator as one of the three pillars LogPulse is built on, not a bolt-on, not a power-user feature, but the foundation everything else stands on.


### Why We Built LogPulse on ClickHouse

*By Gianno Kardjo · March 15, 2026 · 8 min read · Engineering*
*URL: https://logpulse.io/blog/why-we-built-logpulse-on-clickhouse*

When we started building LogPulse, the first architectural decision we had to make was also the most consequential: where do the logs go? The answer to that question would dictate our cost structure, query performance, multi-tenancy model, and ultimately whether we could deliver a product that competes on both price and speed. After months of prototyping, benchmarking, and running production workloads against multiple backends, we chose ClickHouse. This post explains why.

#### The Problem with Elasticsearch

Elasticsearch is the default choice for log analytics, and for good reason. It pioneered full-text search over semi-structured data and built an ecosystem that millions of engineers depend on. But when you operate at scale -- ingesting tens of gigabytes per day across hundreds of tenants -- its limitations become painfully clear.

The JVM heap pressure alone is a constant source of operational toil. Inverted indices consume enormous amounts of memory. Shard management becomes a full-time job. And the storage footprint is staggering: a typical Elasticsearch deployment stores logs at roughly 1.2-1.5x the raw data size after indexing. For a multi-tenant SaaS product where storage costs flow directly to the margin line, that ratio is a dealbreaker.

We needed something that could handle the same query patterns -- keyword search, field filtering, time-range scans, aggregations -- but with fundamentally better economics.

#### Why ClickHouse

ClickHouse is a columnar database originally built at Yandex for web analytics. Its core strengths align almost perfectly with log analytics workloads: it excels at scanning large volumes of time-series data, compresses aggressively, and executes analytical queries with remarkable efficiency.

The compression ratios alone justified the switch. Where Elasticsearch inflates data, ClickHouse compresses it. We consistently see 10-50x compression ratios on real log data depending on the cardinality of the fields. A dataset that would consume 500GB in Elasticsearch fits in 10-50GB in ClickHouse. That is not a rounding error -- it is a fundamental shift in unit economics.

We use the @clickhouse/client library (v0.2.10) for all server-side interactions. The client supports streaming inserts, query cancellation, and session management, which are essential features for a multi-tenant platform where a single runaway query cannot be allowed to affect other users.

#### Schema Design for Multi-Tenancy

Our primary log table lives in the shared_tenants database and uses the MergeTree engine, which is the foundational table engine in ClickHouse. The table is called shared_tenants.logs and its design reflects two competing priorities: query performance for individual tenants and storage efficiency across all tenants.

The partition key is (tenant_id, toYYYYMMDD(timestamp)). This means every tenant-day combination produces a separate set of data parts on disk. When a user queries their logs, ClickHouse prunes all partitions belonging to other tenants before the query even begins executing. This is not a filter applied after a scan -- it is a structural guarantee that tenant data is physically isolated at the storage layer.

The ORDER BY clause is (tenant_id, namespace, timestamp, log_id). This ordering is deliberate. Most queries filter by tenant and namespace first, then scan a time range. By placing these columns at the front of the sort key, ClickHouse can skip entire granules of data that do not match the query predicates. The log_id at the end ensures deterministic ordering for deduplication.

**Partition and sort key**
```sql
-- Partition: physical data isolation per tenant per day
PARTITION BY (tenant_id, toYYYYMMDD(timestamp))

-- Order: optimized for the most common query pattern
ORDER BY (tenant_id, namespace, timestamp, log_id)
```

#### The Column Set

The logs table carries a wide set of columns designed to cover Kubernetes-native logging, structured events, and free-form text. The core columns include tenant_id, timestamp (DateTime64(3) for millisecond precision), log_id, event, level, index, host, source, sourcetype, cluster, namespace, pod, container, and node.

For extensibility, we use three Map(String, String) columns: attributes, parsed_fields, and labels. The Map type in ClickHouse lets us store arbitrary key-value pairs without schema migrations, which is critical when every customer has different log formats. We also store Kubernetes annotations in a dedicated annotations Map column.

On top of these, we define materialized columns -- level_extracted, status_code, and duration_ms -- that are computed at insert time from the raw event data. These columns exist purely for query performance. Instead of parsing a JSON body at query time to extract an HTTP status code, ClickHouse materializes the value once during ingestion and stores it as a native column. Queries that filter on status_code or aggregate by duration_ms skip the parsing step entirely.

#### Full-Text Search with Token Bloom Filters

One of the most common objections to ClickHouse for log analytics is the lack of inverted indices for full-text search. Elasticsearch has had this for over a decade. ClickHouse takes a different approach: token bloom filters.

A token bloom filter is a probabilistic index that can quickly determine whether a data granule does NOT contain a given token. It cannot confirm that a token IS present (false positives are possible), but it can definitively rule out granules that lack the token. For log search, this is exactly the right tradeoff. Most search queries match a tiny fraction of the data. The bloom filter eliminates 95-99% of granules from the scan, and ClickHouse only reads the remaining candidates.

The result is full-text search performance that approaches Elasticsearch for typical queries, with a fraction of the storage and memory overhead.

#### Real-Time Metrics with SummingMergeTree

Beyond raw log search, LogPulse provides real-time dashboards with metrics like log volume, error rates, and latency distributions. Running these aggregations over the raw logs table on every dashboard load would be prohibitively expensive.

Instead, we maintain a separate SummingMergeTree table that pre-aggregates metrics into 5-minute buckets. As logs are ingested, we simultaneously write summary rows that increment counters for each tenant, namespace, and log level. ClickHouse automatically merges these rows in the background, collapsing partial sums into final aggregates. Dashboard queries hit this table instead of the raw logs, returning results in single-digit milliseconds.

#### Query Safety

In a multi-tenant system, you cannot trust user-generated queries. A poorly written aggregation over an unindexed field could consume the entire cluster. We enforce several hard limits: a 30-second execution timeout, a 5GB memory cap per query, and a mandatory tenant_id filter injected by the query compiler. All user-facing queries use parameterized inputs to prevent SQL injection. We also enforce namespace-level RBAC, so users can only query namespaces their role permits.

The default TTL is set to 30 days. Data older than the retention window is automatically dropped by ClickHouse at the partition level, which means no expensive DELETE operations and no tombstone management.

#### Results

After running ClickHouse in production for several months, the numbers speak for themselves. P95 search latency is under 200ms for typical queries across datasets with billions of rows. Storage costs are a fraction of what they would be with Elasticsearch. And the operational surface area is dramatically smaller -- no JVM tuning, no shard rebalancing, no segment merging.

ClickHouse is not without its tradeoffs. It is less mature for full-text search than Elasticsearch, its ecosystem is smaller, and some query patterns (particularly high-cardinality GROUP BY with low selectivity) require careful schema design. But for log analytics at scale, the combination of columnar compression, partition pruning, materialized columns, and bloom filter indices makes it the right foundation for LogPulse.


### Introducing Visual ETL Pipelines

*By Gianno Kardjo · March 8, 2026 · 5 min read · Product*
*URL: https://logpulse.io/blog/introducing-visual-etl-pipelines*

Today we are launching Visual ETL Pipelines, a drag-and-drop pipeline editor that lets you build, test, and deploy log transformation workflows without writing code. If you have ever wrestled with Logstash configuration files, hand-rolled regex parsers, or maintained a brittle chain of shell scripts to clean up your logs before ingestion, this feature was built for you.

#### The Problem

Log data is messy. Application logs, infrastructure metrics, security events, and audit trails all arrive in different formats. Before you can search, alert, or visualize that data, you need to transform it: parse structured fields out of unstructured text, mask sensitive information, enrich events with metadata, and normalize schemas so that logs from different sources can be queried together.

Traditionally, this means writing Logstash pipelines, Fluentd configuration, or custom ETL scripts. These approaches work, but they share a common problem: the transformation logic is buried in configuration files that are difficult to reason about, hard to test, and painful to debug when something breaks at 3am.

#### A Visual Approach

LogPulse Visual ETL Pipelines replace configuration files with a visual canvas. The editor is built on @xyflow/react (ReactFlow), a production-grade library for node-based UIs. You build pipelines by dragging nodes onto the canvas and connecting them with edges. Each node performs a single, well-defined operation on the data flowing through it.

The pipeline engine itself lives in the packages/etl-engine package. When you deploy a pipeline, the engine traverses the node graph and processes each event sequentially through the chain. Every node receives the output of the previous node, transforms it, and passes the result downstream.

#### 18 Node Types Across 6 Categories

We designed the node library to cover the vast majority of log transformation scenarios without requiring custom code. There are 16 node types organized into 6 categories:

Trigger nodes define how a pipeline starts. The start node is the entry point for every pipeline. Extract nodes pull data from external sources -- httpRequest fetches data from APIs, and splunkSearch queries existing Splunk deployments for migration scenarios.

Transform nodes are the core of any pipeline. The transform node applies arbitrary field mappings and transformations. The json and csv nodes parse structured data formats. The redactMask node detects and masks PII fields like email addresses, credit card numbers, and social security numbers -- essential for compliance with data privacy regulations. The fieldOperations node provides fine-grained control over individual fields, and mapCommonSchema normalizes log events into a standard schema.

Flow control nodes handle branching and iteration. The condition node evaluates expressions and routes events to different branches based on the result. The loop, loopStart, loopEnd, and loopBreak nodes let you iterate over arrays within a single event -- useful for processing batch payloads that contain multiple log entries.

Load nodes push processed data to destinations. The logpulseIngest node sends events directly to LogPulse for indexing. The lookup node enriches events by joining against external reference data -- for example, mapping IP addresses to geographic locations or user IDs to team names.

Utility nodes round out the library: log writes diagnostic output for debugging, and error handles failure cases gracefully.

#### Key Nodes in Depth

The redactMask node deserves special attention. When you process logs that may contain PII -- and in practice, almost all logs do -- you need to ensure sensitive data is masked before it reaches your search index. The redactMask node uses pattern matching to identify common PII formats and replaces them with masked values. You configure which patterns to match and how aggressively to mask, and the node handles the rest. This is not a nice-to-have; for teams operating under GDPR, HIPAA, or SOC 2, it is a requirement.

The condition node enables branching logic. You define an expression -- for example, "level equals error" or "status_code greater than 499" -- and the node routes events to different downstream paths based on the evaluation result. This lets you build pipelines that handle different log types differently: errors get enriched with stack trace parsing, warnings get sampled, and debug logs get dropped entirely.

The lookup node connects your pipeline to external data sources for enrichment. You provide a reference dataset (a CSV file, an API endpoint, or another LogPulse index), define the join key, and the node adds matching fields to each event as it passes through.

#### Trigger Modes

Pipelines can be triggered in three ways. Manual triggers let you run a pipeline on demand from the UI, which is ideal for testing and one-off migrations. Scheduled triggers run pipelines on a cron schedule -- for example, every 5 minutes or once a day. Webhook triggers expose an HTTP endpoint that external systems can POST to, enabling real-time ingestion from services that support webhook delivery.

#### A Real-World Example

Consider a common scenario: your application emits JSON-formatted access logs that contain user email addresses in the request body. You need to parse the JSON, mask the email addresses for compliance, enrich each event with the deployment region from a lookup table, and ingest the result into LogPulse.

In the visual editor, this is a five-node pipeline: start (trigger), json (parse the raw log), redactMask (mask email addresses), lookup (add deployment region), and logpulseIngest (write to LogPulse). You connect the nodes with edges, configure each one through its settings panel, and click deploy. The entire process takes under five minutes.

We believe log transformation should be a visual, testable, and collaborative process -- not a solo exercise in regex debugging. Visual ETL Pipelines are available today for all LogPulse users. Head to the Pipelines tab in your workspace to get started.


### The True Cost of Log Management in 2026

*By Gianno Kardjo · February 28, 2026 · 12 min read · Guides*
*URL: https://logpulse.io/blog/true-cost-of-log-management-2026*

If you manage infrastructure at any meaningful scale, you have probably had the "why is our logging bill so high?" conversation. Log management pricing is one of the most opaque areas in the observability market. Vendors quote per-GB rates that seem reasonable until you realize they apply to indexed data after decompression, or that retention beyond 15 days triggers a separate surcharge, or that the per-host fee you overlooked adds thousands to your monthly bill.

We spent weeks analyzing the real cost of running a 100GB/day log pipeline across the four most common options: Splunk Cloud, Datadog Log Management, self-managed Elasticsearch (ELK), and LogPulse. This post lays out what we found.

#### The Pricing Models

Every major vendor uses a different pricing model, which makes apples-to-apples comparison deliberately difficult.

Splunk Cloud prices primarily on daily ingest volume. Their published rates start around $150/GB/day for cloud workloads, with discounts at higher volumes. But the effective rate depends on your contract tier, retention requirements, and whether you need premium features like federated search or SOAR integration.

Datadog Log Management uses a hybrid model: you pay per ingested GB, per indexed GB, and a per-host infrastructure monitoring fee if you use their agent. The ingestion rate is lower than Splunk, but the indexing fee and host fees add up quickly.

Self-managed ELK (Elasticsearch, Logstash, Kibana) has no licensing cost for the open-source stack, but you pay for the infrastructure to run it and the engineering time to operate it. This is the hidden cost that ELK advocates consistently underestimate.

#### The 100GB/Day Benchmark

Let us run the numbers for a concrete scenario: 100GB of log data per day, 30 days of retention, running on a mid-sized Kubernetes cluster with roughly 50 nodes. This is a typical workload for a Series B startup or a mid-market enterprise division.

#### Splunk Cloud

At 100GB/day with 30-day retention, Splunk Cloud typically lands around $15,000 per month. This includes ingest, indexing, storage, and basic search. Premium add-ons like IT Service Intelligence, Enterprise Security, or extended retention push the total higher. The number sounds large, but Splunk customers at this scale frequently report bills in this range after accounting for all the line items.

The most painful aspect of Splunk pricing is the penalty for going over your daily ingest cap. If your application has a logging spike -- a deployment gone wrong, a retry storm, a debug flag left on -- you either eat the overage charge or you lose data. Neither option is acceptable.

#### Datadog Log Management

Datadog at the same scale comes in around $8,000 per month. This includes log ingestion, indexing for the first 15 days, and the infrastructure agent fees for 50 hosts. Extending retention to 30 days adds a storage surcharge. Datadog is generally cheaper than Splunk, but the complexity of their pricing calculator means most teams underestimate their actual bill by 30-50% during the evaluation phase.

Datadog also applies different rates to "ingested" versus "indexed" logs, and encourages you to use exclusion filters to reduce your indexed volume. This works, but it means you are paying for infrastructure to ingest data that you immediately throw away -- a fundamentally inefficient architecture.

#### Self-Managed ELK

Running your own Elasticsearch cluster for 100GB/day requires serious infrastructure. You need dedicated master nodes, hot data nodes with fast SSDs, warm/cold nodes for older data, plus Logstash or Filebeat for ingestion and Kibana for the UI. A typical production deployment at this scale runs on 10-15 VMs or a dedicated Kubernetes cluster.

The infrastructure cost alone is $3,000-5,000 per month on AWS or GCP. But the real cost is the engineering time to operate it. Elasticsearch clusters require constant attention: shard rebalancing, JVM heap tuning, index lifecycle management, upgrade planning, and incident response when the cluster goes red. Most teams need at least one dedicated engineer spending 25-50% of their time on cluster operations. Factor in that engineering cost and self-managed ELK is rarely as cheap as it appears on a spreadsheet.

#### The Hidden Costs

Beyond the sticker price, there are hidden costs that apply across all traditional solutions.

Overage charges are the most obvious. Splunk and Datadog both penalize you for exceeding your committed ingest volume. This creates "log anxiety" -- a real phenomenon where engineering teams avoid adding logging statements, reduce log verbosity, or drop entire log sources to stay under their daily cap. The operational cost of not having the logs you need during an incident is impossible to quantify, but every on-call engineer has felt it.

Retention surcharges are another hidden cost. Most vendors include 15 days of retention in their base price. Extending to 30, 60, or 90 days triggers additional storage fees that scale linearly with volume. For compliance-heavy industries that require 90+ days of log retention, this can double the monthly bill.

Training and migration costs are often ignored during vendor evaluation. Splunk SPL is a powerful but proprietary query language. Datadog has its own query syntax. Moving between vendors means rewriting every saved search, dashboard, and alert. This lock-in is by design.

#### Why Flat-Rate Pricing Matters

LogPulse takes a fundamentally different approach to pricing. We offer flat-rate plans that include a daily ingest allowance, a retention period, and a team member limit -- with no overage charges, no per-host fees, and no retention surcharges.

Our pricing tiers are designed to be simple and predictable. The Free tier includes 1 GB per day with 30-day retention and 2 team members -- enough to evaluate the product properly on a real workload. The Pro tier covers 10 GB per day with 60-day retention and 10 members. And the Business tier handles 50 GB per day with 90-day retention and unlimited members, and unlocks Security Monitoring (SIEM) and Service Intelligence.

Monthly pricing is €59 for Pro and €199 for Business, with custom pricing for Enterprise. There are no hidden fees, no per-GB surcharges, and no penalties for ingest spikes within your tier. You know exactly what you will pay every month, which means your finance team can budget for observability without building in a 50% contingency for overages.

#### The ClickHouse Advantage

How can we offer flat-rate pricing at these levels? The answer is ClickHouse. Because ClickHouse compresses log data at 10-50x ratios compared to raw size, our storage costs per GB ingested are a fraction of what Elasticsearch-based solutions pay. A workload that requires 3TB of Elasticsearch storage fits in 60-300GB of ClickHouse storage. That compression advantage flows directly into our pricing.

Combined with partition-level TTL (30 days default), efficient columnar scans, and materialized columns that eliminate query-time parsing, we can serve the same query patterns at dramatically lower infrastructure cost. We pass those savings on as flat-rate pricing.

#### Getting Started

If you are currently spending more than you think you should on log management, start with our Free tier: 1 GB per day with 30-day retention. That is enough data to ingest logs from a staging environment or a single production service and evaluate whether LogPulse meets your needs. No credit card required, no sales call, no commitment.

The observability market has operated on the assumption that log management must be expensive. We believe the underlying technology has caught up to a point where that is no longer true. Flat-rate pricing is not a gimmick -- it is the natural result of building on a more efficient storage engine.


### LPQL: A Query Language Designed for Speed

*By Gianno Kardjo · February 20, 2026 · 10 min read · Engineering*
*URL: https://logpulse.io/blog/lpql-a-query-language-designed-for-speed*

When we designed the query interface for LogPulse, we had a clear set of requirements. The syntax had to be familiar to engineers who have used Splunk SPL or similar pipe-based query languages. The queries had to compile to efficient ClickHouse SQL, not just work but run fast across billions of rows. And the system had to be secure by default -- no SQL injection, no cross-tenant data leaks, no unbounded resource consumption. The result is LPQL: LogPulse Query Language.

#### Design Goals

LPQL is a pipe-based query language. You start with a search expression, then chain transformations using the pipe operator. Each pipe command takes the output of the previous command and produces a new result set. If you have written SPL, the syntax will feel immediately familiar.

But under the surface, LPQL is fundamentally different from SPL. Every LPQL query is compiled to a ClickHouse SQL statement before execution. The compilation step is not a simple string substitution -- it is a full parse-and-transform pipeline that produces optimized, parameterized SQL with mandatory security constraints.

**LPQL example**
```lpql
search index=web level=error
| where status_code >= 500
| stats count, avg(duration_ms) by host
| sort -count
| head 20
```

#### Architecture: From Text to SQL

The LPQL processing pipeline has four stages: tokenization, parsing, AST construction, and SQL compilation.

The tokenizer breaks the raw query string into tokens: keywords, operators, field names, values, pipes, and literals. It handles quoted strings, escape sequences, and the various comparison operators.

The parser is a recursive descent parser. We chose recursive descent over parser generators like PEG or ANTLR because it gives us precise control over error messages and recovery behavior. When a user writes a malformed query, we want to point to the exact character where parsing failed and suggest a fix. Generated parsers make this kind of error reporting difficult.

The parser produces an abstract syntax tree (AST) that represents the query as a tree of typed nodes. Each pipe command becomes a node in the AST, with its arguments and modifiers attached as child nodes. The AST is the canonical representation of the query -- all subsequent processing operates on the tree, not the raw text.

The SQL compiler walks the AST and emits a ClickHouse SQL statement. Each pipe command corresponds to a SQL transformation. A search becomes a WHERE clause. A stats command becomes a GROUP BY with aggregate functions. A sort becomes an ORDER BY. The compiler nests these transformations as subqueries, building up the final SQL from the inside out.

#### The 22 Pipe Commands

LPQL supports 22 pipe commands that cover search, transformation, aggregation, and formatting.

Search and filtering: search performs keyword and field matching, where applies arbitrary boolean expressions, dedup removes duplicate events by field value, and fields selects or removes specific fields from the output.

Aggregation: stats computes aggregate functions over groups, timechart produces time-bucketed aggregations for charting, chart creates pivot-table style aggregations, top and rare find the most and least common values for a field.

Transformation: eval creates computed fields using expressions, rename changes field names, rex extracts fields using regular expressions, bin (aliased as bucket) discretizes numeric or time values into fixed-width intervals, fillnull replaces null values with a default, and filldown propagates the last non-null value forward.

Flow control: table formats output as a flat table, head and tail limit results to the first or last N rows, sort orders results by one or more fields, and join combines results from two searches.

Utility: makeresults generates synthetic events for testing queries without hitting real data.

**Aggregation with timechart**
```lpql
search index=api
| timechart span=5m count by level
| fillnull value=0
```

#### How Compilation Works

Each pipe command in the AST is compiled to a SQL subquery transformation. The compiler maintains a running query context and wraps each new pipe as an outer query around the previous result. Consider this simple LPQL query:

**LPQL input**
```lpql
search index=web level=error
| stats count, avg(duration_ms) as avg_duration by host
| sort -avg_duration
| head 10
```

The compiler produces SQL that looks conceptually like the following (simplified for clarity):

**Compiled ClickHouse SQL (simplified)**
```sql
SELECT *
FROM (
  SELECT host,
         count() AS count,
         avg(duration_ms) AS avg_duration
  FROM shared_tenants.logs
  WHERE tenant_id = {tenant_id:String}
    AND index = {p0:String}
    AND level = {p1:String}
  GROUP BY host
)
ORDER BY avg_duration DESC
LIMIT 10
```

Notice several things about this output. The tenant_id filter is injected automatically -- the user never specifies it, and they cannot remove it. The literal values for index and level are passed as parameterized query arguments ({p0:String}, {p1:String}), not interpolated into the SQL string. And the subquery nesting follows the pipe order: search produces the innermost query, stats wraps it with GROUP BY, sort adds ORDER BY, and head adds LIMIT.

#### Security by Default

Security is not a feature we bolted on after building the compiler. It is baked into the architecture at every level.

Every query gets a mandatory tenant_id filter. The compiler injects this predicate into the innermost WHERE clause, and it cannot be overridden or removed by the user. This ensures that ClickHouse partition pruning eliminates all data belonging to other tenants before any computation begins.

Namespace RBAC adds a second layer. Each user has a set of permitted namespaces based on their role. The compiler adds a namespace IN (...) predicate that restricts results to only the namespaces the user is authorized to access.

All literal values are passed as parameterized query arguments. This eliminates SQL injection entirely -- there is no code path where user-provided text is concatenated into the SQL string.

On the resource side, we enforce a 100K result limit on all queries to prevent memory exhaustion on the client. Queries are capped at a 50-condition limit to prevent adversarial query complexity. And ClickHouse enforces a 30-second execution timeout and a 5GB memory cap per query at the server level.

#### Performance: Why LPQL Queries Are Fast

LPQL query performance comes from the combination of the compiler optimizations and ClickHouse schema features.

Materialized columns are the biggest single optimization. When a query filters on level, status_code, or duration_ms, it reads from pre-computed columns instead of parsing the event body. This turns what would be a full-text scan into a simple column comparison, which ClickHouse executes orders of magnitude faster.

Token bloom filters accelerate full-text search terms. When the compiler encounters a keyword search (a bare string not attached to a specific field), it generates a hasToken() predicate that leverages the bloom filter index. ClickHouse skips entire data granules that the bloom filter identifies as non-matching.

Partition pruning ensures that every query starts by eliminating irrelevant partitions. The mandatory tenant_id filter prunes all other tenants. Time range predicates prune all non-matching days. By the time ClickHouse begins scanning data, it is typically reading less than 1% of the total table.

#### Stats Functions

The stats, timechart, and chart commands support a comprehensive set of aggregate functions: count for event counts, dc (distinct count) for cardinality, avg, sum, min, max for basic arithmetic aggregates, median and mode for distribution analysis, stdev for standard deviation, perc50 and p95 for percentile calculations, values and list for collecting distinct or all values into arrays, and first and last for positional aggregates.

These functions compile directly to ClickHouse aggregate functions. For example, dc compiles to uniqExact(), perc50 compiles to quantileExact(0.5)(), and values compiles to groupUniqArray(). The mapping is designed to produce the most efficient ClickHouse function for each semantic operation.

**Stats functions in practice**
```lpql
search index=api
| stats count, dc(user_id) as unique_users,
        avg(duration_ms) as avg_latency,
        p95(duration_ms) as p95_latency,
        values(host) as hosts
  by namespace
```

#### What Comes Next

LPQL is under active development. We are working on sub-search support for correlated queries, lookup command integration for enrichment at query time, and a query optimizer that can reorder pipe commands for better partition pruning. The goal is a query language that makes ClickHouse feel as natural for log analysis as SPL made Elasticsearch feel a decade ago -- but with better performance, stronger security, and no vendor lock-in.


### Building LogPulse Without a US Kill Switch

*By Gianno Kardjo · March 22, 2026 · 11 min read · Engineering*
*URL: https://logpulse.io/blog/building-logpulse-without-a-us-kill-switch*

In March 2025, a US executive order threatened to ban European companies from using American cloud services unless certain data-sharing conditions were met. The order was walked back within days, but the message was clear: if your product runs on American infrastructure, your business exists at the pleasure of American trade policy. We started building LogPulse six months later, and that episode shaped every architectural decision we made.

This post is not about politics. It is about engineering. Specifically, it is about how we designed a production-grade log management platform where every core component is either European, open-source, or replaceable within a week. Not as a theoretical exercise -- as a concrete, tested migration path.

#### The Principle: Core vs. Peripheral

We draw a hard line between core and peripheral components. Core components are the ones your customers interact with directly: the database that stores their logs, the query engine that searches them, the API that ingests them, the pipeline engine that transforms them. If a core component disappears overnight, your product is dead.

Peripheral components support the product but do not define it: the CDN that serves the marketing site, the email service that sends password reset links, the OAuth provider that offers social login, the AI model that generates query suggestions. If a peripheral component disappears, you have a bad week -- but your customers can still ship logs, search them, and set up alerts.

Our rule is simple: core components must be either European or open-source with no single-vendor dependency. Peripheral components can be American, but each one must have a documented European alternative and a migration plan that takes no more than five working days.

#### The Core Stack: No American Dependencies

Let us walk through the core stack and explain why each choice was made.

#### ClickHouse: Log Storage

ClickHouse is the foundation of LogPulse. It stores every log entry, executes every search query, and powers every dashboard. We wrote an entire blog post about why we chose it over Elasticsearch, but the sovereignty angle deserves its own explanation.

ClickHouse is open-source under the Apache 2.0 license. It was originally developed at Yandex in Russia and is now maintained by ClickHouse Inc. The source code is publicly available and can be self-hosted on any infrastructure. We run ClickHouse Cloud on GCP europe-west4 (Netherlands) -- our log data physically resides in a European data center and never leaves the EU.

If ClickHouse Inc. ceased to exist tomorrow, we could self-host the same software on Hetzner bare-metal servers in Falkenstein, Germany. The @clickhouse/client library we use is open-source. The schema, the queries, the materialized columns -- everything is portable. There is no proprietary API, no vendor lock-in, no license key that can be revoked.

#### PostgreSQL: Metadata & Configuration

All non-log data -- user accounts, team configurations, API keys, alert rules, pipeline definitions -- lives in PostgreSQL. Like ClickHouse, PostgreSQL is open-source with no single-vendor dependency. We use Drizzle ORM for migrations and type-safe queries, but Drizzle generates standard SQL. If we needed to switch ORMs, the database schema would remain unchanged.

PostgreSQL runs on any cloud provider, any VPS, any bare-metal server. Moving it from a US-hosted provider to a European one is a pg_dump and pg_restore away.

#### Redis: Job Queue

Our log ingestion pipeline uses BullMQ backed by Redis for job queuing. Both are open-source. Redis runs anywhere. The queue configuration is fewer than 50 lines of code. Migrating to a European-hosted Redis instance means changing a connection string.

#### Fastify: API Framework

The LogPulse API runs on Fastify, an open-source Node.js web framework primarily maintained by an Italian and European core team. It is not owned by a US company, not governed by a US foundation, and not dependent on any proprietary service. Our entire API -- ingestion, querying, authentication, pipeline management -- runs on Fastify with standard open-source plugins for CORS, rate limiting, WebSocket support, and security headers.

#### React Flow: Pipeline Editor

The visual ETL pipeline editor uses React Flow (now @xyflow/react), built and maintained by a team in Germany. The library is open-source under the MIT license. It is one of the most critical UI components in the product -- users build their entire data transformation logic on it -- and it has zero American corporate dependency.

#### LPQL: Our Own Query Language

We built LPQL (LogPulse Query Language) from scratch. The tokenizer, parser, AST, and SQL compiler all live in our own packages/lpql-parser package. There is no external dependency for query processing. LPQL compiles directly to ClickHouse SQL, and since ClickHouse is open-source, the entire query pipeline from user input to database result is free of proprietary dependencies.

This is a deliberate contrast to products that depend on Elasticsearch Query DSL or Splunk SPL. If those vendors change their query interface, their customers have no recourse. LPQL is ours, and it goes wherever we go.

#### The Peripheral Stack: American, but Replaceable

We use several American services for peripheral functionality. Here is each one, why we chose it, and what the migration path looks like.

#### OpenAI → Mistral AI

The AI Query Generator uses OpenAI gpt-4.1-mini to translate natural language into LPQL queries. This is a convenience feature -- it makes LogPulse easier to use, but it is not required. Every query the AI generates can be written manually by the user.

The migration path is straightforward. Mistral AI is a French company with models that match or exceed GPT-4.1-mini performance for structured output tasks. Our AI integration is a single file (ai-query-generator.ts) with a clean interface: natural language in, LPQL out. Swapping the model provider means changing the API endpoint and adapting the prompt format. We estimate two days of work, including testing.

Alternatively, we could self-host an open-source model like Mistral or Llama using vLLM on European GPU infrastructure. This eliminates the external dependency entirely.

#### Cloudflare → Hetzner or BunnyCDN

Cloudflare serves our marketing website (logpulse.io) and runs a small community API on Workers. The marketing site is a static React app -- it can be served from any CDN or web server. The community API is fewer than 200 lines of code and could run on any edge platform or a simple Express server.

BunnyCDN is a Slovenian company with European-owned infrastructure. Hetzner, based in Germany, offers cloud servers and object storage at competitive prices. Moving the static site to either takes an afternoon. The community API could move to a Hetzner VPS or a Fly.io instance in Amsterdam.

#### Railway → Hetzner Cloud or OVH

Our API server and background worker currently run on Railway, a US-based platform. Railway is convenient for deployment, but our services are standard Node.js applications with no Railway-specific code. They use environment variables for configuration and Docker for containerization.

Migrating to Hetzner Cloud (Germany) or OVH (France) means deploying the same Docker containers to European infrastructure. We use no Railway-specific APIs, no proprietary build hooks, no platform-locked features. The deployment script would change; the application code would not.

#### Microsoft Azure Email → Brevo

Password reset emails and alert notifications currently route through Microsoft Graph API via Azure AD. This is arguably the easiest component to replace. Brevo (formerly Sendinblue) is a French email platform with full EU data residency. Our email service is a single package (packages/email) with a clean send() interface. Swapping the transport layer from Microsoft Graph to Brevo SMTP means changing one file.

#### Google & GitHub OAuth → Self-Hosted

We offer Google and GitHub as social login options. These are convenience features -- users can always sign up with email and password. If we needed to remove US OAuth providers, we could replace them with European alternatives or deploy a self-hosted Keycloak instance. Our authentication library (Better Auth) supports arbitrary OAuth providers through a plugin interface.

Better Auth itself is open-source and runs entirely within our API process. There is no external auth service to depend on -- the library handles session management, password hashing, and token generation locally.

#### The Migration Playbook

We maintain an internal document that describes, for every American dependency, the exact steps to migrate to a European alternative. Here is the summary:

Bedrock to Mistral AI: swap API client and prompt format in a single file. Estimated effort: 2 days. Log data exposure: the AI features read log data to answer questions, so a swap moves that processing to the new provider; inference today runs on Claude via Amazon Bedrock in eu-north-1, inside the EU.

Cloudflare to BunnyCDN or Hetzner: redeploy static assets and update DNS. Estimated effort: 1 day. Community API to Hetzner VPS: containerize and deploy. Estimated effort: 1 day.

Railway to Hetzner Cloud: deploy existing Docker containers to Hetzner. Migrate PostgreSQL via pg_dump. Point Redis to Hetzner-hosted instance. Estimated effort: 3 days.

Azure Email to Brevo: change email transport in packages/email. Estimated effort: half a day.

Google and GitHub OAuth to Keycloak: deploy Keycloak on European infrastructure, update OAuth configuration. Estimated effort: 3 days.

Total migration from fully American peripheral stack to fully European: approximately 10 working days. During migration, core functionality (log ingestion, search, alerts, pipelines) continues to operate without interruption because none of those systems depend on American services.

#### What About React, TypeScript, and Node.js?

Yes, React is maintained by Meta. TypeScript is maintained by Microsoft. Node.js is stewarded by the OpenJS Foundation. These are all open-source projects licensed under permissive terms (MIT and Apache 2.0). They cannot be revoked, restricted, or turned off. The source code is mirrored on thousands of servers worldwide. No trade policy can prevent you from running npm install.

There is a meaningful difference between depending on a US SaaS service (where access can be terminated with a configuration change) and using open-source software created by a US company (where the code is permanently in the public domain). We care about the former. The latter is a non-issue.

#### Why This Matters

European companies processing log data have regulatory obligations under GDPR. Many operate in sectors -- healthcare, finance, government -- where data sovereignty is not a preference but a legal requirement. Even outside regulated industries, the past two years have demonstrated that geopolitical risk is operational risk.

We built LogPulse so that a European company can adopt it knowing that their logs are stored in Europe, processed by open-source software, and served by infrastructure that can be relocated to fully European providers within two weeks. Not as a future roadmap item -- as a tested, documented capability.

Your log data is too important to exist at the mercy of a foreign trade policy. We designed LogPulse so it never has to.


### Why We Put AI at the Heart of LogPulse

*By Gianno Kardjo · May 6, 2026 · 9 min read · Product*
*URL: https://logpulse.io/blog/why-we-put-ai-at-the-heart-of-logpulse*

Most log platforms in 2026 still optimise for the same workflow that existed in 2010: a human types a query, scans a result list, refines the query, repeats. The interface gets faster, the storage gets cheaper, the syntax gets prettier, but the cognitive load on the engineer at 3am is unchanged. We do not believe that workflow is the destination. We believe it is the floor. AI is what raises the ceiling, and that is why we built LogPulse with AI as a first-class component of the platform, not an autocomplete bolted onto the search bar.

This post explains where AI is today inside LogPulse, why we chose root cause analysis as the first surface to invest in, and where this is going. It is also a deliberate look forward: the long-term thesis is that the same primitives that make a useful Investigator also make a SIEM that catches threats no signature was ever written for.

#### The Problem with "Search Faster"

When something breaks in production, the engineer on call does not have a search problem. They have an explanation problem. They need to know what changed, what failed, what cascaded, and which of the seventeen alerts firing simultaneously is the cause and which are the consequences. Faster queries help, but the bottleneck is rarely the database. It is the time spent forming hypotheses, running them, reading the results, and forming the next one.

Every observability vendor has spent years optimising the wrong half of that loop. The query runs in 200ms; the human still takes 45 minutes to chain enough queries together to land on the answer. Improving query speed by another 10x changes nothing if the bottleneck is human reasoning time.

AI is the first technology in two decades that meaningfully attacks the second half of the loop. Not because it knows the answer up front, but because it can run dozens of hypotheses in parallel, read the evidence, and narrate its reasoning in a way the engineer can audit and correct.

#### What the Investigator Actually Does

The LogPulse AI Investigator is a streaming agent that runs on Anthropic Claude. When you ask it a question -- "why is the checkout API returning 500s in production?" -- it does not generate a single LPQL query and call it a day. It runs an iterative loop: form a hypothesis, call a tool, read the result, refine, call another tool, until it has enough evidence to synthesise a structured answer.

Under the hood, the investigator has access to a deliberately constrained tool surface: LPQL search, time-bucketed aggregation, field-context lookups, system-health snapshots, and the ability to spawn sub-investigations for parallel branches of inquiry. It runs in a streaming loop with extended thinking turned on, so the UI can show the engineer the model's reasoning as it happens. When the model emits multiple tool calls in a single turn, they fan out via Promise.allSettled and return concurrently.

The output is not a paragraph of prose. It is a structured Finding[] array, where every finding carries a title, a severity, an array of supporting evidence (the actual log lines, with their timestamps), and a confidence score. The web UI renders each finding as a clickable card that drills into the underlying logs, so the engineer can verify the claim before acting on it. This is deliberate -- AI that cannot be audited is AI that cannot be trusted in production.

**Finding shape: what the Investigator emits**
```typescript
interface Finding {
  title: string;            // human-readable claim
  severity: 'low' | 'medium' | 'high' | 'critical';
  evidence: Evidence[];     // actual log lines with timestamps
  confidence: number;       // 0..1, surfaced in the UI
}
```

#### Why Root Cause First

When we sat down to decide where AI should land first, we ranked features by two axes: how much value AI adds beyond what a faster query produces, and how forgiving the surface is when the model is wrong. Root cause analysis scored highest on both.

On the first axis, the lift is enormous. The investigator is doing the work the on-call engineer would otherwise do manually -- scanning related services, correlating timestamps, forming hypotheses, ruling them out. A model that runs that loop in two minutes instead of forty-five is genuinely useful, even when it gets some of the intermediate steps wrong, because the engineer is still in the loop and the evidence is auditable.

On the second axis, the blast radius of a wrong answer is small. The engineer reads the finding, looks at the evidence, and either agrees or does not. The Investigator never autonomously takes action on infrastructure. This is in deliberate contrast to surfaces where AI has been pushed prematurely: auto-remediation, auto-scaling decisions, auto-triage that closes tickets without review. Those surfaces punish wrong answers severely. Root cause analysis does not.

Starting with root cause also gives us the right learning signal. Every investigation produces feedback: explicit thumbs-up/thumbs-down, but also implicit signals like which findings the engineer clicked into and which they ignored. We feed positive-feedback investigations into a pattern memory store, keyed by a Haiku-extracted pattern signature. Subsequent investigations in the same organisation get matching insights injected into the system prompt. The platform learns the shape of your incidents over time, without ever training a model on your data.

#### The Second AI Surface: Co-Building Pipelines

Root cause analysis is one shape AI takes inside LogPulse. Building data pipelines is another. The Pipeline AI is a chat agent embedded directly in the visual ETL editor: same Claude backend, same auditable approach, different problem surface. Where the Investigator answers "why is this broken", the Pipeline Agent answers "how do I get this data in, in the shape I actually need it".

Building an ETL pipeline is a hypothesis-and-evidence loop too. You sketch a graph, run it, look at what failed, adjust. The bottleneck for most users is not the editor itself. It is knowing what an integration's auth header looks like, which node parses what, which sequence of operations turns the raw API response into a row that lands in the right index. New users hit a documentation wall before they hit the actual work. The Pipeline Agent is built to walk through that wall with them.

The agent sits in a side panel next to the canvas. It can read the current pipeline graph, the last run's output per node, the structured run logs that the engine writes into the tenant's internal indexes, and historical versions of the same pipeline for diff. It can also list and read other pipelines in the workspace as references. And it can mutate the canvas: add a node, insert a node between two existing ones, update a node's configuration, connect or disconnect edges, remove things. Every change it makes shows up on the canvas with a coloured "added" or "modified" badge, so you can see at a glance what the agent touched and audit each change before promoting the draft.

For integrations the model does not reliably know (Datadog's logs-ingest endpoint, Stripe's pagination cursor shape, Auth0's audit-log payload format) it can look up the official documentation via Anthropic-hosted web search, capped at three lookups per turn. And critically: it asks you first. The web-search protocol is not a model quirk; it is an explicit UX rule. Web pages can contain prompt-injection attempts dressed up as instructions, and we treat any third-party content as untrusted. So the agent surfaces a one-line confirmation -- "I'd like to look up the Datadog logs-API docs to confirm the auth header. OK to search?" -- and only searches once you say yes. Tenant secrets, API keys, payload values, and any other data from your environment are never put into a search query.

The result is a workflow where you describe what you want in plain language and the agent walks you through making it real. A common shape:

**A typical Pipeline Agent session**
```text
you: every hour, pull our staging error logs from the last hour
     and POST a markdown digest to this Slack webhook: https://...

agent: I'd like to look up the Slack incoming-webhook docs to confirm
       the request shape. OK to search?

you: yes

agent: [web_search → reads Slack docs]
       Slack expects a POST with a JSON body { "text": "..." } and a
       2xx response counts as delivered. Building this as four nodes:

         start → lpqlSearch → transform → httpRequest

       - lpqlSearch runs `level=error earliest=-1h` against your tenant
         logs and lands the rows on the payload.
       - transform folds the rows into a markdown summary string.
       - httpRequest POSTs { text: "<summary>" } to your webhook.

       I'll set the trigger to schedule every hour. Sound right?

you: yes, do it

agent: [add_node × 3, add_edge × 3, configures each node]
       Done. Three nodes badged "added" on the canvas. Want me to run
       it once now to verify the Slack delivery before we promote?
```

Most pipelines that fit this shape come together in a few minutes. The bulk of that time is the conversation, not the typing, and the conversation is where the value is, because the agent is asking the questions that catch the mistakes a copy-pasted template would have shipped silently. "Are you sure you want this triggered every minute and not every hour?" "This webhook URL looks like it includes a secret. Should I store it in pipeline secrets instead of inline?" "You're reading from index=apps and ingesting back into index=apps. That loops. Did you mean a different destination?"

The constraints are deliberate. The agent only mutates the current pipeline's draft, never historical versions and never another pipeline (it can read those for reference, but editing them is a UI action you take, not the agent). It refuses bulk-destructive actions (wiping all nodes, removing every edge) without explicit confirmation. And the node library it can build with is exactly the eighteen node types the engine actually supports, because the system prompt is grounded in the same NODE_DOCS reference a human reads in our docs. The agent cannot invent a node that does not exist; it has to compose what is there.

The Investigator and the Pipeline Agent are the same architectural pattern applied to two different problems. A constrained tool surface, an auditable action trail (badges on changed nodes, evidence on findings), a model that prefers to ask one clarifying question rather than guess, and an explicit consent rule before any operation that touches third-party content. That pattern is what makes both surfaces production-grade rather than demo-grade, and it is the foundation we are building everything else on top of.

#### Why This Is Step One, Not the Destination

The Investigator and the Pipeline Agent are the first two surfaces where we are willing to put AI in front of a paying customer, because the value is high and the failure mode is benign. But the deeper bet is that the same primitives -- a constrained tool surface, a structured output schema, an auditable evidence trail, pattern memory, an explicit consent rule before third-party content enters the conversation -- generalise to a problem that the security industry has been failing at for thirty years: detecting threats that no one has written a signature for.

Modern SIEMs are still fundamentally signature engines. They detect what someone has already seen and codified. They miss novel attack chains, lateral movement that does not match a rule, insider misuse that looks superficially normal. The industry has spent two decades writing more rules and calling it progress. The result is alert fatigue, false-positive rates that train SOC analysts to ignore the tool, and detection coverage that stops at the boundary of what the rule writers anticipated.

An AI agent that can iterate over evidence, form hypotheses, and produce structured findings with audit trails is the right shape for security detection too. The Investigator we ship today asks "why is this service erroring?" The Investigator we are building toward asks "is anything unusual happening across this organisation's logs in the last hour?", and answers it without requiring someone to have written a rule for that specific anomaly first.

We are not there yet. We have shipped the foundation: pattern detection via Drain3-style log templating, statistical anomaly detection, and the agent infrastructure that can reason over both. The hard work ahead is the security-specific tooling, the threat-model coverage, and the operational maturity that turns "an LLM noticed something weird" into "a SOC analyst can act on this with the same confidence they would act on a Sigma rule." That is a multi-year arc. Root cause analysis is how we prove the technology and earn the right to take it there.

#### The Macro: Every Organisation Is Figuring AI Out

Outside our product, the same conversation is happening in every enterprise we talk to. Heads of platform are standing up internal AI provisions. Data teams are being formed or restructured to own model evaluation, prompt management, and the guardrails that keep AI usage compliant. CIOs are deciding which models to centralise on, which providers to allow, and how to expose those provisions to product teams without losing control.

This is not hype-cycle activity. It is real org-design work, with budget attached, and it is happening in roughly every Series-C-and-above company in Europe right now. The teams driving it have a problem we recognise: they have invested in the AI capability, and now they need products to plug into it that respect the boundaries they have built. Not products that ship a hardcoded provider, log everything to a vendor cloud, and wave away the data-residency conversation.

When those teams ask us for choice, the request usually breaks down into three concrete reasons. The first is data sovereignty. Dutch and broader European organisations are facing tighter requirements from GDPR, BIO, and the EU AI Act. For municipalities, hospitals, water authorities, and regulated financial institutions, "where does my log data live during AI processing" is no longer an academic question. It is a compliance line item. An American AI provider may be excellent technically, but excellence does not override jurisdiction. The second is vendor lock-in. Mature IT organisations refuse to take a critical dependency on a single provider whose pricing, terms, or product roadmap can change unilaterally. The third is leverage on existing investment. Customers who already operate an Azure OpenAI tenant, a GPU cluster running vLLM, or a self-hosted Mistral or Llama deployment want that investment reflected in the tools they buy, not duplicated alongside it.

LogPulse is built to plug in. The AI module lives behind a clean interface: prompt blocks composed in code, tool schemas defined in one file, a single Anthropic client today that reads ANTHROPIC_API_KEY. Everything that talks to a model goes through it. Swapping the provider, changing the model tier, routing through an organisation-internal AI gateway -- these are configuration shapes, not architectural rewrites. The structured-finding output schema, the security guardrails, the evidence-rendering UI work the same regardless of where the actual inference happens.

We are honest about today: production LogPulse runs on managed Claude through Anthropic's API, in Europe-resident infrastructure where the model provider supports it. We do not currently offer a customer-facing "bring your own model" toggle, but that is the next step on this trajectory, not a maybe.

#### Three Deployment Shapes, One Product

The architecture we are building toward is one product with three deployment shapes, picked per customer based on what they actually need. Each shape is the right answer for a different operational reality.

The first shape is managed cloud AI. This is what every customer uses today. The Investigator runs against managed Claude via Anthropic's API, in EU-resident infrastructure, and we operate the entire pipeline. Setup time is zero, model performance is the strongest currently available, and the customer trades a small amount of architectural control for a lot of operational simplicity. For most teams, this is exactly the right trade.

The second shape is EU-hosted open-source inference, operated by us. For customers where data residency is a hard compliance requirement and a US-headquartered model provider is not acceptable regardless of where the metal lives, we are building toward AI features that run end-to-end on open-weight models hosted on European GPU infrastructure under our operational umbrella. Same Investigator UX, same structured-finding output, different inference backend. This is the shape that makes the public-sector and regulated-finance conversation possible without forcing the customer to operate their own GPUs.

The third shape is bring-your-own-endpoint. Customers who already run an Azure OpenAI tenant, an internal AI gateway, or a self-hosted Mistral, Llama, or vLLM deployment can point LogPulse at that endpoint and use it for inference. The customer keeps full control over which model is called, where the inference runs, what gets logged, and what gets billed. We run the product; they run the model. This is also the shape that scales to airgapped on-premises, where the AI endpoint is the customer's own GPUs inside their own perimeter and no traffic ever leaves their network.

None of these three is shipping in the trial today beyond the first. We are saying out loud that all three are the direction. We would rather have customers make the cloud-AI choice today knowing the on-prem shape is on the path, than have them adopt us today and discover later that the architecture cannot follow them where they need to go.

#### How We Get There

The non-AI parts of LogPulse are already structured for self-hosted deployment. ClickHouse, PostgreSQL, Redis, Fastify, React Flow. Every core component is open-source and runs anywhere a Kubernetes cluster does. The same containers that run our managed cloud run on a customer's infrastructure. We have written about this elsewhere: the kill-switch post lays out which dependencies are core and which are peripheral, and how long it takes to migrate each one.

The AI side is the part that needs new work. The first milestone is provider abstraction at the AI client layer, a clean dispatcher so the same Investigator code can target managed Claude, an EU-hosted open-weight model on our infrastructure, or a customer-supplied endpoint, through one interface. The Investigator's prompts, tool schemas, and finding shapes are designed to be portable, but model-specific behaviour (cache breakpoints, structured-output reliability, tool-use parallelism) varies enough that this is real engineering, not a configuration flip.

After that comes the EU-hosted tier: selecting and operating an open-weight model that meets the structured-output and tool-use bar the Investigator depends on, on European GPU infrastructure we run. Then the on-prem package: a Helm chart, a documented set of model requirements (so customers know what hardware they need to host their own endpoint), and a deployment guide for customers who want to run the entire stack inside their own perimeter, including airgapped.

Enterprise customers in regulated sectors (defence, critical infrastructure, healthcare networks under NIS2, public-sector bodies operating under BIO) have been telling us they need this for as long as we have been talking to them. They are right. The principle is simple: a customer should never have to choose between good AI features and control over their data. Cloud-only is not a viable answer for every customer, and pretending otherwise would be dishonest.

#### What We Are Asking You to Trust

The honest version of this post is that we are asking you to trust two things. First, that AI applied carefully to the right surface -- root cause analysis, with auditable evidence and a human in the loop -- is genuinely useful today, not a demo-quality novelty. We have shipped it, customers use it, and the structured-finding format is what makes it production-grade rather than a chatbot.

Second, that the foundation we are building on -- the agent loop, the tool surface, the pattern memory, the multi-provider architecture -- is the right shape for the next set of problems. SIEM that detects beyond signatures. EU-hosted inference for residency-bound customers. Bring-your-own-endpoint for organisations that have already invested in their own AI provision. Airgapped on-prem for the most demanding deployments.

Both of those bets are still bets. The Investigator gets things wrong sometimes; we tell you when it does and we use the feedback to make it better. The SIEM trajectory is multi-year work. The EU-hosted and bring-your-own-endpoint tiers are on the roadmap, not in the trial today. We would rather be straight about what is shipped, what is in flight, and what is further out, than oversell. The customers who choose us based on a clear-eyed picture of where we are tend to be the customers we want.

If you are running an organisation that is figuring out how to put AI to work, and you want a logging platform built to plug into that effort rather than duplicate it, talk to us. The AI Investigator and the Pipeline Agent are in every plan today. The conversation about which deployment shape is right for you -- managed cloud, EU-hosted, or your own endpoint -- is the conversation we want to be having with enterprise customers over the next year.