Send Google Cloud logs to LogPulse
Google Cloud logs from a Pub/Sub subscription that a log sink writes to: Cloud Audit Logs, VPC flow and firewall logs, Security Command Center findings and other log entries.
Configure → Sources → Cloud. LogPulse reads the Pub/Sub pull subscription itself, without an agent. Events land in the audit, network, firewall, application indexes with source google_cloud and fill the Change, Network Traffic, Alerts data models.Overview
- Pub/Sub pull subscription to Integration poller
- Integration poller to index=audit
- Integration poller to index=network
- Integration poller to index=firewall
- Integration poller to index=application
LogPulse signs in to Google Cloud with the credential you give it and fetches new events every five minutes by default. You can set the interval per integration, from one minute to 24 hours.
What you need
- A log sink to a Pub/Sub topic with a pull subscription
- A service account with Pub/Sub Subscriber on it
- In LogPulse: a role that may manage API keys. Cloud integrations fall under that right.
1. Create a credential in Google Cloud
| Field in LogPulse | Where you find it |
|---|---|
| Service account key (JSON) | A service account with the Pub/Sub Subscriber role on the subscription. In Google Cloud: IAM, Service accounts, Keys, Add key (JSON). Stored encrypted and never shown again. |
| Pub/Sub subscription | Create a log sink (Logging, Log Router) to a Pub/Sub topic, at project, folder or organization level, and a pull subscription on that topic: projects/<project>/subscriptions/<name>. |
2. Add the integration in LogPulse
Go to Configure → Sources → Cloud and choose Google Cloud. Fill in the fields, tick the data sources you want and save. LogPulse first tests the connection with your credential and tells you what is wrong when it fails.
What arrives
| Data source | Index | Source | Sourcetype | What it holds |
|---|---|---|---|---|
| Admin Activity audit logs | audit | google_cloud | google:gcp:pubsub:audit:admin_activity | Every change to resources and IAM: who, which method, on which resource, from where, and whether it was allowed. |
| System Event audit logs | audit | google_cloud | google:gcp:pubsub:audit:system_event | Changes Google made itself, such as live migration and automatic restarts. |
| Data Access audit logs (off by default) | audit | google_cloud | google:gcp:pubsub:audit:data_access | Reads of data and metadata. Only when switched on in the project; high volume. |
| Security Command Center | audit | google_cloud | google:gcp:security:alerts | Findings from a Security Command Center notification: category, severity, state and resource. |
| VPC flow logs (off by default) | network | google_cloud | google:gcp:compute:vpc_flows | Network flows of VMs and GKE nodes: source, destination, ports, bytes. |
| Firewall rules logging | firewall | google_cloud | google:gcp:compute:firewall | Connections a VPC firewall rule allowed or denied. |
| Other log entries (off by default) | application | google_cloud | google:gcp:pubsub:message | Every other log entry in the subscription, such as application and GKE logs. |
The index is the kind of data and the source is the product, so detection rules for that kind of data find Google Cloud without changes.
Fields and data models
| Data model | What you get |
|---|---|
| Change | Audit log entries with principal, method, resource and caller IP. |
| Network Traffic | Flows and firewall decisions with source, destination and ports. |
| Alerts | Security Command Center findings with category and severity. |
Field names follow the CIM field names. These are the fields LogPulse sets on Google Cloud events; a field is left out when Google Cloud did not send a value.
accountactionbytes_outcategorydelegateddescriptiondestdest_ipdest_namedest_portdest_typedirectionhttp_user_agentidlog_namemessageobjectpackets_outreasonregionruleseveritysignaturesrcsrc_ipsrc_namesrc_portstatustransportuservendor_actionvpc
The original record of Google Cloud is kept with each event under raw, so nothing the vendor sent is lost.
3. Check the data
The first events arrive within a few minutes. Check what came in:
earliest=-1h index IN (audit, network, firewall, application) source IN (google_cloud)
| stats count by index, source, sourcetypeThe integration page in LogPulse shows when it last read events and the last error, if any.
Troubleshooting
The connection test names what is wrong. These are the answers you are most likely to see.
| What you see | What to do |
|---|---|
| projects/acme-prod/subscriptions/logpulse does not exist. | Check the project and the name of the subscription. It must be a pull subscription on the topic the log sink writes to. |
| The service account needs the Pub/Sub Subscriber role on this subscription (and Viewer to see it). | Grant the service account the roles Pub/Sub Subscriber and Pub/Sub Viewer on the subscription. |
| Google refused the service account key. Check that the key is active. | The key was deleted or the service account is disabled. Create a new JSON key. |
| The test passes but no events arrive. | Run the search above over a longer time. No results: check on the integration page that the data source is switched on and when it last read events. Google Cloud may also simply have had nothing new. |
Frequently asked questions
- How do I send Google Cloud logs to LogPulse?
- Add the Google Cloud cloud integration under Configure, Sources, Cloud in LogPulse and fill in Service account key (JSON), Pub/Sub subscription. LogPulse reads the Pub/Sub pull subscription itself, every five minutes by default; there is nothing to install.
- Which permissions does LogPulse need in Google Cloud?
- A log sink to a Pub/Sub topic with a pull subscription. A service account with Pub/Sub Subscriber on it. Read access is enough for collecting logs.
- Which index does Google Cloud data go to?
- The index is the kind of data: audit, network, firewall, application. The source is the product: google_cloud. Each data source has its own sourcetype, listed on this page.
- Which data models does Google Cloud fill?
- Change, Network Traffic, Alerts. Detection rules and dashboards written for a data model work on Google Cloud without changes.
- Why is Data Access audit logs and VPC flow logs and Other log entries off by default?
- To keep your daily ingest down until you need it: Data Access audit logs, VPC flow logs, Other log entries are high volume or of use to fewer teams. Tick the data source in the integration to switch it on.
- What happens to log entries of a data source I switched off?
- They are acknowledged and not kept, so the subscription does not fill up and they do not count towards your volume. Narrow the filter of the log sink to stop paying Google for them.
- Can one integration read every Google Cloud project?
- Yes. Create the log sink on the organization or a folder, with child resources included, and it brings the logs of all projects to one topic.