Send AWS logs in S3 logs to LogPulse
CloudTrail, GuardDuty, Security Hub, WAF, VPC Flow Logs, load balancer, CloudFront and S3 access logs from your log buckets, read once per file through an SQS queue.
Configure → Sources → Cloud. LogPulse reads the S3 bucket notifications on an SQS queue itself, without an agent. Events land in the audit, web, network indexes with source aws and fill the Cloud Audit, Alert, Web Access, Network Activity data models.Overview
- S3 bucket notifications on an SQS queue to Integration poller
- Integration poller to index=audit
- Integration poller to index=web
- Integration poller to index=network
LogPulse signs in to AWS logs in S3 with the credential you give it and fetches new events every five minutes by default. You can set the interval per integration, from one minute to 24 hours.
What you need
- The log bucket(s) sending "All object create events" to an SQS queue (via SNS or EventBridge is fine)
- A key that may read and delete messages on the queue and read the objects (and decrypt them for SSE-KMS)
- For Security Hub and GuardDuty events: an EventBridge rule to a Firehose stream that writes to the bucket
- In LogPulse: a role that may manage API keys. Cloud integrations fall under that right.
1. Create a credential in AWS logs in S3
| Field in LogPulse | Where you find it |
|---|---|
| SQS queue URL | A queue that gets the "object created" notifications of your log bucket(s), directly, through SNS or through EventBridge. |
| Access key ID | An IAM user or role key with sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the queue, s3:GetObject on the log bucket(s), and kms:Decrypt when the bucket uses SSE-KMS. |
| Secret access key | Stored encrypted and never shown again. |
2. Add the integration in LogPulse
Go to Configure → Sources → Cloud and choose AWS logs in S3. Fill in the fields, tick the data sources you want and save. LogPulse first tests the connection with your credential and tells you what is wrong when it fails.
What arrives
| Data source | Index | Source | Sourcetype | What it holds |
|---|---|---|---|---|
| CloudTrail | audit | aws | aws:cloudtrail | Management and data events of a trail or an organization trail: every account and region it covers. |
| GuardDuty findings | audit | aws | aws:guardduty | Findings GuardDuty exports to S3, or GuardDuty events from EventBridge through Firehose. |
| Security Hub findings | audit | aws | aws:securityhub:finding | Findings from Security Hub (Inspector, Macie, Config, the security standards) from EventBridge through Firehose. |
| AWS WAF | web | aws | aws:waf | Web ACL logs: client IP, request, matched rule and action. High volume when every request is logged. |
| VPC Flow Logs (off by default) | network | aws | aws:vpcflow | Accepted and rejected connections per network interface. High volume. |
| Load balancer access logs (off by default) | web | aws | aws:elb:accesslogs | Application and Classic Load Balancer requests. |
| CloudFront standard logs (off by default) | web | aws | aws:cloudfront:accesslogs | Requests to your CloudFront distributions. |
| S3 server access logs (off by default) | audit | aws | aws:s3:accesslogs | Requests to S3 buckets: requester, operation, key and status. |
The index is the kind of data and the source is the product, so detection rules for that kind of data find AWS logs in S3 without changes.
Fields and data models
| Data model | What you get |
|---|---|
| Cloud Audit | CloudTrail with the same fields as the CloudTrail API connector, so the AWS rules fire on both. |
| Alert | GuardDuty and Security Hub findings in the shared alert shape. |
| Web Access | WAF, load balancer and CloudFront requests with src_ip, http_method, url, status and user_agent. |
| Network Activity | VPC flow records with the 5-tuple and allow or deny. |
Events are mapped to the data models above, with the same field names as other sources of the same kind. See field names.
The original record of AWS logs in S3 is kept with each event under raw, so nothing the vendor sent is lost.
3. Check the data
The first events arrive within a few minutes. Check what came in:
earliest=-1h index IN (audit, web, network) source IN (aws)
| stats count by index, source, sourcetypeThe integration page in LogPulse shows when it last read events and the last error, if any.
Troubleshooting
The connection test names what is wrong. These are the answers you are most likely to see.
| What you see | What to do |
|---|---|
| AWS refused the request. The key needs sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on this queue, and s3:GetObject on the log bucket(s). | Add those permissions to the IAM user or role of the key. |
| Queue readable, no notifications waiting yet. | Check that the log bucket sends "All object create events" to this queue, directly, through SNS or through EventBridge. |
| The queue fills up but no logs arrive. | Reading the files is checked on the first sync: the key also needs s3:GetObject on the log bucket, and kms:Decrypt when the bucket uses SSE-KMS. |
| The test passes but no events arrive. | Run the search above over a longer time. No results: check on the integration page that the data source is switched on and when it last read events. AWS logs in S3 may also simply have had nothing new. |
Frequently asked questions
- How do I send AWS logs in S3 logs to LogPulse?
- Add the AWS logs in S3 cloud integration under Configure, Sources, Cloud in LogPulse and fill in SQS queue URL, Access key ID, Secret access key. LogPulse reads the S3 bucket notifications on an SQS queue itself, every five minutes by default; there is nothing to install.
- Which permissions does LogPulse need in AWS logs in S3?
- The log bucket(s) sending "All object create events" to an SQS queue (via SNS or EventBridge is fine). A key that may read and delete messages on the queue and read the objects (and decrypt them for SSE-KMS). For Security Hub and GuardDuty events: an EventBridge rule to a Firehose stream that writes to the bucket. Read access is enough for collecting logs.
- Which index does AWS logs in S3 data go to?
- The index is the kind of data: audit, web, network. The source is the product: aws. Each data source has its own sourcetype, listed on this page.
- Which data models does AWS logs in S3 fill?
- Cloud Audit, Alert, Web Access, Network Activity. Detection rules and dashboards written for a data model work on AWS logs in S3 without changes.
- Why is a data source off by default?
- To keep your daily ingest down until you need it: VPC Flow Logs, Load balancer access logs, CloudFront standard logs, S3 server access logs are high volume or of use to fewer teams. Tick the data source in the integration to switch it on.
- Can one queue serve several AWS accounts?
- Yes. One queue can take the notifications of several buckets and of an organization trail, so every account and region comes in through one integration.