Send Google Cloud logs to LogPulse

Google Cloud logs from a Pub/Sub subscription that a log sink writes to: Cloud Audit Logs, VPC flow and firewall logs, Security Command Center findings and other log entries.

In short
To send Google Cloud logs to LogPulse, add Google Cloud under Configure → Sources → Cloud. LogPulse reads the Pub/Sub pull subscription itself, without an agent. Events land in the audit, network, firewall, application indexes with source google_cloud and fill the Change, Network Traffic, Alerts data models.

Overview

How Google Cloud data reaches LogPulse
Vendor
Pub/Sub pull subscription
Collect
Integration pollerreads new events, maps the fields
Store
index=audit
index=network
index=firewall
index=application
  1. Pub/Sub pull subscription to Integration poller
  2. Integration poller to index=audit
  3. Integration poller to index=network
  4. Integration poller to index=firewall
  5. Integration poller to index=application

LogPulse signs in to Google Cloud with the credential you give it and fetches new events every five minutes by default. You can set the interval per integration, from one minute to 24 hours.

What you need

  • A log sink to a Pub/Sub topic with a pull subscription
  • A service account with Pub/Sub Subscriber on it
  • In LogPulse: a role that may manage API keys. Cloud integrations fall under that right.

1. Create a credential in Google Cloud

Field in LogPulseWhere you find it
Service account key (JSON)A service account with the Pub/Sub Subscriber role on the subscription. In Google Cloud: IAM, Service accounts, Keys, Add key (JSON). Stored encrypted and never shown again.
Pub/Sub subscriptionCreate a log sink (Logging, Log Router) to a Pub/Sub topic, at project, folder or organization level, and a pull subscription on that topic: projects/<project>/subscriptions/<name>.

2. Add the integration in LogPulse

Go to Configure → Sources → Cloud and choose Google Cloud. Fill in the fields, tick the data sources you want and save. LogPulse first tests the connection with your credential and tells you what is wrong when it fails.

Tip: Secrets are stored encrypted and never shown again. To change one, paste a new value; the rest stays.

What arrives

Data sourceIndexSourceSourcetypeWhat it holds
Admin Activity audit logsauditgoogle_cloudgoogle:gcp:pubsub:audit:admin_activityEvery change to resources and IAM: who, which method, on which resource, from where, and whether it was allowed.
System Event audit logsauditgoogle_cloudgoogle:gcp:pubsub:audit:system_eventChanges Google made itself, such as live migration and automatic restarts.
Data Access audit logs (off by default)auditgoogle_cloudgoogle:gcp:pubsub:audit:data_accessReads of data and metadata. Only when switched on in the project; high volume.
Security Command Centerauditgoogle_cloudgoogle:gcp:security:alertsFindings from a Security Command Center notification: category, severity, state and resource.
VPC flow logs (off by default)networkgoogle_cloudgoogle:gcp:compute:vpc_flowsNetwork flows of VMs and GKE nodes: source, destination, ports, bytes.
Firewall rules loggingfirewallgoogle_cloudgoogle:gcp:compute:firewallConnections a VPC firewall rule allowed or denied.
Other log entries (off by default)applicationgoogle_cloudgoogle:gcp:pubsub:messageEvery other log entry in the subscription, such as application and GKE logs.

The index is the kind of data and the source is the product, so detection rules for that kind of data find Google Cloud without changes.

Note: LogPulse reads up to 5,000 entries per round and acknowledges what it read. Entries of a data source you switched off are acknowledged and not kept.
Note: A sink on the organization brings the logs of all projects in through one integration.

Fields and data models

Data modelWhat you get
ChangeAudit log entries with principal, method, resource and caller IP.
Network TrafficFlows and firewall decisions with source, destination and ports.
AlertsSecurity Command Center findings with category and severity.

Field names follow the CIM field names. These are the fields LogPulse sets on Google Cloud events; a field is left out when Google Cloud did not send a value.

  • account
  • action
  • bytes_out
  • category
  • delegated
  • description
  • dest
  • dest_ip
  • dest_name
  • dest_port
  • dest_type
  • direction
  • http_user_agent
  • id
  • log_name
  • message
  • object
  • packets_out
  • reason
  • region
  • rule
  • severity
  • signature
  • src
  • src_ip
  • src_name
  • src_port
  • status
  • transport
  • user
  • vendor_action
  • vpc

The original record of Google Cloud is kept with each event under raw, so nothing the vendor sent is lost.

3. Check the data

The first events arrive within a few minutes. Check what came in:

earliest=-1h index IN (audit, network, firewall, application) source IN (google_cloud)
| stats count by index, source, sourcetype

The integration page in LogPulse shows when it last read events and the last error, if any.

Troubleshooting

The connection test names what is wrong. These are the answers you are most likely to see.

What you seeWhat to do
projects/acme-prod/subscriptions/logpulse does not exist.Check the project and the name of the subscription. It must be a pull subscription on the topic the log sink writes to.
The service account needs the Pub/Sub Subscriber role on this subscription (and Viewer to see it).Grant the service account the roles Pub/Sub Subscriber and Pub/Sub Viewer on the subscription.
Google refused the service account key. Check that the key is active.The key was deleted or the service account is disabled. Create a new JSON key.
The test passes but no events arrive.Run the search above over a longer time. No results: check on the integration page that the data source is switched on and when it last read events. Google Cloud may also simply have had nothing new.

Frequently asked questions

How do I send Google Cloud logs to LogPulse?
Add the Google Cloud cloud integration under Configure, Sources, Cloud in LogPulse and fill in Service account key (JSON), Pub/Sub subscription. LogPulse reads the Pub/Sub pull subscription itself, every five minutes by default; there is nothing to install.
Which permissions does LogPulse need in Google Cloud?
A log sink to a Pub/Sub topic with a pull subscription. A service account with Pub/Sub Subscriber on it. Read access is enough for collecting logs.
Which index does Google Cloud data go to?
The index is the kind of data: audit, network, firewall, application. The source is the product: google_cloud. Each data source has its own sourcetype, listed on this page.
Which data models does Google Cloud fill?
Change, Network Traffic, Alerts. Detection rules and dashboards written for a data model work on Google Cloud without changes.
Why is Data Access audit logs and VPC flow logs and Other log entries off by default?
To keep your daily ingest down until you need it: Data Access audit logs, VPC flow logs, Other log entries are high volume or of use to fewer teams. Tick the data source in the integration to switch it on.
What happens to log entries of a data source I switched off?
They are acknowledged and not kept, so the subscription does not fill up and they do not count towards your volume. Narrow the filter of the log sink to stop paying Google for them.
Can one integration read every Google Cloud project?
Yes. Create the log sink on the organization or a folder, with child resources included, and it brings the logs of all projects to one topic.