The hot tier stores the most recent logs on NVMe SSDs with full ClickHouse columnar indexing. Queries against hot data benefit from primary key indexes, skip indexes, and in-memory caching. This tier handles the vast majority of interactive queries, live tail streams, and alert evaluations.

Hot tier capacity is determined by your plan. When hot storage is full, the oldest data is automatically migrated to the warm tier. The migration process runs continuously in the background without impacting query performance.

The warm tier uses high-capacity HDDs with ZSTD compression. Data is reorganized into larger, sequential file blocks optimized for range scans. Warm tier queries are slower than hot tier but significantly faster than cold tier, making this tier suitable for incident investigations that span multiple days.

The cold tier stores data in columnar Parquet format on object storage (Amazon S3, Google Cloud Storage, or Azure Blob Storage). Queries against cold data are executed on-demand using ephemeral compute instances. Cold queries have higher latency but are billed only for the data scanned, making this tier cost-effective for infrequent access patterns.

Cold data can be rehydrated to the warm tier on demand for faster access during extended investigations. Rehydration typically completes within 5-15 minutes depending on the volume of data.

You can set a custom retention period on any index, overriding the plan default. Custom retention can be shorter or longer than the plan default, within the maximum allowed by your plan.

curl -X PUT https://api.logpulse.io/api/v1/indexes/web-access-logs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LOGPULSE_API_KEY" \
  -d '{
    "retention": {
      "hot_days": 3,
      "warm_days": 14,
      "cold_days": 60,
      "total_days": 77
    }
  }'

An index is a logical grouping of logs with shared retention policies and access controls. By default, all logs are stored in the "default" index. Create additional indexes to separate logs by environment, service, team, or compliance requirement.

curl -X POST https://api.logpulse.io/api/v1/indexes \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LOGPULSE_API_KEY" \
  -d '{
    "name": "security-audit",
    "description": "Security and audit trail logs",
    "retention": {
      "total_days": 90
    }
  }'

Index patterns route incoming logs to the correct index based on source, level, or attribute values. Patterns are evaluated in priority order and the first match determines the target index.

# Route security logs to the security-audit index
Pattern: source=auth-service OR source=firewall OR level=security
Target: security-audit
Priority: 1

# Route debug logs to a short-retention index
Pattern: level=debug OR level=trace
Target: debug-logs
Priority: 2

# Everything else goes to the default index
Pattern: *
Target: default
Priority: 100

Each index can have its own retention policy that overrides the organization default. This allows you to keep security audit logs for 90 days while deleting debug logs after 3 days, all within the same LogPulse organization.

Consider an application generating 50 GB/day of raw logs with a 90-day retention policy:

Raw ingestion:        50 GB/day
Compression ratio:    10:1
Stored per day:       5 GB/day

Hot storage (14 days):
  5 GB/day x 14 days = 70 GB
  70 GB x $0.80/GB   = $56.00/month

Warm storage (days 15-30):
  5 GB/day x 16 days = 80 GB
  80 GB x $0.25/GB   = $20.00/month

Cold storage (days 31-90):
  5 GB/day x 60 days = 300 GB
  300 GB x $0.03/GB  = $9.00/month

Ingestion:
  50 GB/day x 30 days x $0.10/GB = $150.00/month

Total estimated monthly cost: $235.00/month

Scheduled exports run automatically at a configured interval and export logs matching a LPQL query to the target destination. Exports can run hourly, daily, or weekly.

curl -X POST https://api.logpulse.io/api/v1/exports \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LOGPULSE_API_KEY" \
  -d '{
    "name": "daily-security-export",
    "query": "index=security-audit",
    "schedule": "daily",
    "format": "parquet",
    "destination": {
      "type": "s3",
      "bucket": "company-log-archive",
      "prefix": "logpulse/security/",
      "region": "us-east-1",
      "role_arn": "arn:aws:iam::123456789012:role/logpulse-export"
    },
    "compression": "zstd",
    "partition_by": ["date", "source"]
  }'

Exported files are partitioned by date and optionally by additional fields (source, level, etc.) for efficient querying with tools like Athena, BigQuery, or Spark.

To comply with GDPR right-to-erasure requests, LogPulse supports targeted deletion of logs containing specific personal data. Use the data deletion API to remove logs matching a query:

curl -X POST https://api.logpulse.io/api/v1/data/delete \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LOGPULSE_API_KEY" \
  -d '{
    "query": "attributes.user_id=\"usr_deleted_user\"",
    "reason": "GDPR right-to-erasure request #12345",
    "dry_run": false
  }'

SOC 2 Type II audits typically require an audit trail retained for at least 1 year. LogPulse audit logs (user actions, API key usage, configuration changes) are automatically retained for 90 days on current plans (up to 365 days on Enterprise, coming soon). For longer retention, configure archival exports to your own storage. Application logs used for security monitoring should use the maximum retention available on your plan.

For organizations handling protected health information (PHI), LogPulse offers HIPAA-eligible configurations on the Enterprise plan. This includes encryption at rest (AES-256), encryption in transit (TLS 1.2+), audit logging of all data access, and a signed Business Associate Agreement (BAA).

HIPAA requires a minimum 6-year retention period for certain records. Configure indexes containing PHI-related logs with a 2190-day (6-year) retention policy or set up archival exports to your own HIPAA-compliant storage.

Log volume depends on your application architecture, log verbosity, and traffic patterns. Here are typical ranges by source type:

LogPulse applies ZSTD compression to all stored data. Typical compression ratios vary by log format:

For planning purposes, use a conservative 10:1 compression ratio. Actual ratios may be higher, especially for highly repetitive log formats like access logs.

The Storage Dashboard (Settings, then Storage) shows real-time and historical storage usage broken down by index and tier. Key metrics include total stored volume, daily ingestion rate, storage growth trend, and projected time until quota is reached.

LogPulse automatically sends notifications when your storage usage approaches plan limits:

The Storage Dashboard includes automated cleanup recommendations based on your usage patterns:

Reduce debug log retention. If debug or trace level logs consume more than 30% of your storage, consider reducing their retention to 1-3 days or filtering them out during ingestion using ETL pipelines.

Archive before deletion. For indexes approaching retention expiry, configure archival exports to preserve data in your own storage at a lower cost.

Drop unused fields. Large attribute payloads increase storage consumption. Use ETL pipelines to drop fields that are never searched or filtered.

Consolidate duplicate logs. If multiple sources emit the same log event (for example, sidecar and application both logging the same request), deduplicate at the ingestion layer.