GCP

The GCP scrapers scrapes your GCP account to fetch all the resources & save them as configs.

gcp-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: gcp-flanksource
  namespace: mc
spec:
  gcp:
    # An organization on its own scrapes every project beneath it. Add projects to
    # narrow it to those that belong to the organization. Listing projects without
    # an organization still works, but identities are then tenanted by project.
    #- organization: "1234567890"
    #  projects:
    #    - workload-prod-eu-02
    - project: workload-prod-eu-02
      exclude:
        - SecurityCenter
        #- IAMGroupMembers  # disable Google-group expansion (needs Cloud Identity groups.readonly)
      #connection: connection://mc/gcloud-flanksource
      # IAMPolicy and IAMGroupMembers run by default only when include is empty.
      # Once include filters asset types, list the IAM flags explicitly to keep IAM data:
      #include:
        #- storage.googleapis.com/Bucket
        #- container.googleapis.com/Cluster
        #- IAMPolicy         # RBAC: users/groups/service-accounts -> roles
        #- IAMGroupMembers   # expand Google group membership
        # AuditLogs is opt-in and only runs when listed here:
        #- AuditLogs         # access history from the BigQuery audit-log dataset
      #auditLogs:
        #dataset: default._AllLogs
        # Project holding the dataset. Required when scraping an organization or
        # more than one project, since the dataset lives in exactly one project.
        #project: logging-prod
        #since: 30d
        #excludeMethods:
          #- io.k8s.*

Scraper

Field	Description	Scheme	Required
logLevel	Specify the level of logging.	string
schedule	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	string
retention	Settings for retaining changes, analysis and scraped items	Retention
gcp	GCP scrape config	[]GCP

GCP

note

Either the connection name or the credentials are required (if Workload Identity is not being used)

Field	Description	Scheme
project*	GCP Project ID	string
auditLogs	Query BigQuery dataset for audit logs	AuditLogs
connection	The connection url to use, mutually exclusive with credentials	Connection
credentials	The credentials to use for authentication	EnvVar
endpoint	Custom GCP Endpoint to use	string
exclude	GCP resources to exclude from scraping	[]string
include	GCP resources to include for scraping	[]string
skipTLSVerify	Skip TLS verification when connecting to GCP	boolean
labels	Labels for each config item.	map[string]string
properties	Custom templatable properties for the scraped config items.	[]ConfigProperty
tags	Tags for each config item. Max allowed: 5	[]ConfigTag
transform	Transform configs after they've been scraped	Transform

Audit Logs

Field	Description	Scheme
dataset	BigQuery dataset to query audit logs from (e.g., "default._AllLogs")	string
since	Time range to query audit logs (e.g., "24h", "7d", "30d")	string
userAgents	Filter user agents matching these patterns	MatchExpressions
principalEmails	Filter principal emails matching these patterns	MatchExpressions
permissions	Filter permissions matching these patterns	MatchExpressions
serviceNames	Filter service names matching these patterns	MatchExpressions
methods	Filter methods matching these patterns	MatchExpressions