Logs Action

The Logs action fetches and queries logs from various backends. Use it to retrieve logs for debugging, analysis, or to pass to downstream actions like AI analysis.

kubernetes-logs.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: kubernetes-logs
  namespace: mc
spec:
  title: Kubernetes Logs
  icon: logs
  category: Logs
  description: Fetch logs from Kubernetes
  configs:
    - types:
        - Kubernetes::Pod
        - Kubernetes::Deployment
        - Kubernetes::StatefulSet
        - Kubernetes::DaemonSet
  parameters:
    - name: limit
      label: Limit
      description: The maximum number of logs to fetch
      required: false
      default: "100"
  actions:
    - name: Fetch logs from Loki
      logs:
        kubernetes:
          kind: $(.config.config_class)
          apiVersion: $(.config.config.apiVersion)
          namespace: $(.config.tags.namespace)
          name: $(.config.name)
          limit: $(.params.limit)
          start: now-2h

Field	Description	Scheme
`name*`	Step Name	`string`
`logs`	Logs Action	Logs
`delay`	A delay before running the action e.g. `8h`	`Duration` or CEL with Playbook Context
`filter`	Conditionally run an action	CEL with Playbook Context
`runsOn`	Which runner (agent) to run the action on	`[]Agent`
`templatesOn`	Where templating (and secret management) of actions should occur	`host` or `agent`
`timeout`	Timeout on this action.	Duration

Logs

Specify exactly one backend. All of the post processing fields are set inside the backend, alongside its query.

Field	Description	Scheme
`cloudwatch`	Query logs from AWS CloudWatch Logs	CloudWatch
`kubernetes`	Fetch logs directly from Kubernetes pods	Kubernetes
`loki`	Query logs from Grafana Loki	Loki
`opensearch`	Query logs from OpenSearch/Elasticsearch	OpenSearch

Post processing

Every backend accepts the following fields to filter, parse and deduplicate logs after they have been retrieved.

Field	Description	Scheme
`dedupe.fields`	Fields to use for identifying duplicate log entries (e.g., `['message', 'level']`). Two logs that have an empty value for a field are still deduped	`[]string`
`dedupe.window`	Time window for deduplication (e.g., `5m`, `1h`). Logs with identical dedupe fields within this window are merged	Duration
`mapping.dedupBy`	Fields to dedupe the returned logs by	`[]string`
`mapping.groupBy`	Fields to group the returned logs by	`[]string`
`mapping.host`	Source field names for the host	`[]string`
`mapping.id`	Source field names for the unique log identifier	`[]string`
`mapping.ignore`	Fields to drop from the log labels	`[]string`
`mapping.message`	Source field names for the log message content	`[]string`
`mapping.severity`	Source field names for the log level/severity	`[]string`
`mapping.source`	Source field names for the log source	`[]string`
`mapping.timestamp`	Source field names for the timestamp (tries each until non-empty)	`[]string`
`match`	CEL expressions to filter logs after retrieval. A log is kept if any of the expressions match	[]MatchExpression
`parse`	Log format to parse the message with	`logfmt` \| `klogfmt` \| `json` \| `syslog` \| `autodetect`

Time range

The cloudwatch, kubernetes and loki backends share a common time range and limit.

Field	Description	Scheme
`end`	End time of the query. Defaults to now	`string`
`limit`	Maximum number of log lines to return	`string`
`start`	Start time of the query. Supports datemath e.g. `now-24h`	`string`

CloudWatch

Query logs from AWS CloudWatch Logs using CloudWatch Logs Insights.

Field	Description	Scheme
`logGroup*`	CloudWatch log group to query	`string`
`query*`	CloudWatch Logs Insights query to run on the log group	`string`
`accessKey`	AWS access key ID	EnvVar
`assumeRole`	ARN of the role to assume	`string`
`connection`	AWS connection for credentials	AWSConnection
`endpoint`	Custom AWS endpoint	`string`
`region`	AWS region (e.g., `us-east-1`)	`string`
`secretKey`	AWS secret access key	EnvVar
`sessionToken`	AWS session token	EnvVar
`skipTLSVerify`	Skip TLS verification when connecting to AWS	`boolean`

Example

cloudwatch.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: cloudwatch-events
  namespace: mc
spec:
  title: Cloudwatch Events
  icon: cloudwatch
  category: Logs
  description: Fetch events from Cloudwatch
  parameters:
    - name: logGroup
      label: Log Group
      description: The log group to fetch events from
      required: true
    - name: limit
      label: Limit
      description: The maximum number of events to fetch
      required: false
      default: '100'
  configs:
    - types:
        - AWS::::Account
  actions:
    - name: Fetch events from CloudWatch
      logs:
        cloudwatch:
          start: now-24h
          limit: $(.params.limit)
          logGroup: $(.params.logGroup)
          query: |
            fields @message, @timestamp, @logStream, @log
            | sort @timestamp desc
            | limit $(.params.limit)

Kubernetes

Fetch logs directly from Kubernetes pods using the Kubernetes API. Logs are fetched for the resource identified by kind, apiVersion, namespace and name — for a workload such as a Deployment, logs from all of its pods are returned.

Field	Description	Scheme
`apiVersion`	API version of the resource	`string`
`connection`	Kubernetes connection for cluster access	KubernetesConnection
`containers`	Fetch logs from only these containers	[]MatchExpression
`kind`	Kind of the resource to fetch logs for e.g. `Pod`, `Deployment`, `StatefulSet`, `DaemonSet`	`string`
`kubeconfig`	Kubeconfig content or path	EnvVar
`name`	Name of the resource	`string`
`namespace`	Namespace of the resource	`string`
`pods`	Include pods matching any of these selectors. Applies when fetching logs at a higher resource level, such as a deployment spanning multiple pods	[]ResourceSelector

Example

k8s.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: kubernetes-logs
  namespace: mc
spec:
  title: Kubernetes Logs
  icon: logs
  category: Logs
  description: Fetch logs from Kubernetes
  configs:
    - types:
        - Kubernetes::Pod
        - Kubernetes::Deployment
        - Kubernetes::StatefulSet
        - Kubernetes::DaemonSet
  parameters:
    - name: limit
      label: Limit
      description: The maximum number of logs to fetch
      required: false
      default: "100"
  actions:
    - name: Fetch logs from Loki
      logs:
        kubernetes:
          kind: $(.config.config_class)
          apiVersion: $(.config.config.apiVersion)
          namespace: $(.config.tags.namespace)
          name: $(.config.name)
          limit: $(.params.limit)
          start: now-2h

Loki

Query logs from Grafana Loki using LogQL.

Field	Description	Scheme
`query*`	LogQL query (e.g., `{app="myservice"} \|= "error"`)	`string`
`connection`	Connection name for Loki credentials	Connection
`direction`	Query direction: `forward` (oldest first) or `backward` (newest first)	`forward` \| `backward`
`interval`	Only return entries at or greater than this interval	`string`
`password`	Basic auth password	EnvVar
`since`	Duration used to calculate `start` relative to `end`. Any value set for `start` supersedes this	`string`
`step`	Query resolution step width, as a duration or a number of seconds	`string`
`url`	Loki server URL (e.g., `http://loki:3100`)	`string`
`username`	Basic auth username	EnvVar

Example

loki.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: loki-logs
  namespace: mc
spec:
  title: Loki Logs
  icon: logs
  category: Logs
  description: Fetch logs from Loki
  configs:
    - types:
        - Kubernetes::Pod
        - Kubernetes::Deployment
  parameters:
    - name: limit
      label: Limit
      description: The maximum number of logs to fetch
      required: false
      default: "100"
  actions:
    - name: Fetch logs from Loki
      logs:
        loki:
          url: https://logs-prod-111.grafana.net
          username:
            valueFrom:
              secretKeyRef:
                name: loki-grafana-cloud
                key: userid
          password:
            valueFrom:
              secretKeyRef:
                name: loki-grafana-cloud
                key: password
          query: |
            {namespace="{{ .config.tags.namespace }}",{{.config.config_class | toLower}}="{{ .config.name }}"}
          limit: $(.params.limit)
          start: now-2h
          match:
            - severity != "info" && severity != "unknown"
            - labels.service == 'payment'
          dedupe:
            window: 1h
            fields:
              - message

OpenSearch

Query logs from OpenSearch or Elasticsearch clusters.

Field	Description	Scheme
`index*`	Index name or pattern (e.g., `logs-`, `app-logs-2024.01.`)	`string`
`query*`	OpenSearch query DSL	`string`
`address`	OpenSearch cluster address	`string`
`connection`	Connection name for OpenSearch credentials	Connection
`limit`	Maximum number of documents to return	`string`
`password`	Basic auth password	EnvVar
`username`	Basic auth username	EnvVar

Example

opensearch.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: opensearch-logs
  namespace: mc
spec:
  title: OpenSearch Logs
  icon: elasticsearch
  category: Logs
  description: Fetch logs from OpenSearch
  configs:
    - types:
        - Kubernetes::Pod
        - Kubernetes::Deployment
  parameters:
    - name: limit
      label: Limit
      description: The maximum number of logs to fetch
      required: false
      default: '100'
  actions:
    - name: Fetch logs from OpenSearch
      logs:
        opensearch:
          address: http://localhost:9200
          query: |
            {
              "query": {
                "bool": {
                  "filter": [
                    { "term": { "kubernetes.namespace": "$(.config.tags.namespace)" } },
                    { "term": { "kubernetes.labels.app": "$(.config.name)" } }
                  ]
                }
              }
            }
          index: k8s-logs
          limit: $(.params.limit)

Output

The action returns a structured result containing the retrieved logs.

Field	Description	Scheme
`groups`	Logs grouped by `mapping.groupBy`, when set	[]LogGroup
`logs`	Log entries with normalized fields	[]LogLine
`metadata`	Backend specific metadata about the query	map[string]any

Log line

Field	Description	Scheme
`count`	Number of occurrences the line was deduped from	`integer`
`firstObserved`	Time the log line was first seen	time
`hash`	Hash of the tokenized message, used to group similar lines	`string`
`host`	Host the log originated from	`string`
`id`	Unique identifier of the log line	`string`
`labels`	Remaining fields from the log line	`[map[string]string]`
`lastObserved`	Time the log line was last seen. Set when the line has been deduped	time
`message`	Log message	`string`
`severity`	Log level	`string`
`source`	Source of the log	`string`

Using Results in Subsequent Actions

actions:
  - name: fetch-logs
    logs:
      kubernetes:
        kind: Deployment
        apiVersion: apps/v1
        namespace: production
        name: api
        start: now-1h
  - name: analyze
    if: 'size(getAction("fetch-logs").result.logs) > 0'
    ai:
      prompt: |
        Analyze these log entries:
        {{.actions.fetch-logs.result.logs | toJSON}}

Templating

CEL Expressions

The following variables can be used within the CEL expressions of filter, if, delays and parameters.default:

Field	Description	Schema
`config`	Config passed to the playbook	`ConfigItem`
`check`	Canary Check passed to the playbook	`Check`
`playbook`	Playbook passed to the playbook	`Playbook`
`run`	Current run	`Run`
`params`	User provided parameters to the playbook	`map[string]any`
`request`	Webhook request	`Webhook Request`
`env`	Environment variables defined on the playbook	`map[string]any`
`user.name`	Name of the user who invoked the action	`string`
`user.email`	Email of the user who invoked the action	`string`
`agent.id`	ID of the agent the resource belongs to.	`string`
`agent.name`	Name of the agent the resource belongs to.	`string`

Conditionally Running Actions

Playbook actions can be selectively executed based on CEL expressions. These expressions must either return

a boolean value (true indicating run the action & skip the action otherwise)
or a special function among the ones listed below

Function	Description
`always()`	run no matter what; even if the playbook is cancelled/fails
`failure()`	run if any of the previous actions failed
`skip()`	skip running this action
`success()`	run only if all previous actions succeeded (default)
`timeout()`	run only if any of the previous actions timed out

delete-kubernetes-pod.yaml
---
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: notify-send-with-filter
spec:
  parameters:
    - name: message
      label: The message for notification
      default: '{{.config.name}}'
  configs:
    - types: 
        - Kubernetes::Pod
  actions:
    - name: Send notification
      exec:
        script: notify-send "{{.config.name}} was created"
    - name: Bad script
      exec:
        script: deltaforce
    - name: Send all success notification
      if: success() # this filter practically skips this action as the second action above always fails
      exec:
        script: notify-send "Everything went successfully"
    - name: Send notification regardless
      if: always()
      exec:
        script: notify-send "a Pod config was created"

Defaulting Parameters

delete-kubernetes-pod.yaml
apiVersion:
  mission-control.flanksource.com/v1
  kind: Playbook
  metadata:
    name: edit
  spec:
    title: 'Edit Kustomize Resource'
    icon: flux
    parameters:
    - default: 'chore: update $(.config.type)/$(.config.name)'
      name: commit_message

Go Templating

When templating actions with Go Templates, the context variables are available as fields of the template's context object . eg .config, .user.email

Templating Actions

delete-kubernetes-pod.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: scale-deployment
spec:
  description: Scale Deployment
  configs:
    - types:
        - Kubernetes::Deployment
  parameters:
    - name: replicas
      label: The new desired number of replicas.
  actions:
    - name: kubectl scale
      exec:
        script: |
          kubectl scale --replicas={{.params.replicas}} \
            --namespace={{.config.tags.namespace}} \
            deployment {{.config.name}}

Functions

Function	Description	Return
`getLastAction()`	Returns the result of the action that just run	Action Specific
`getAction({action})`	Return the result of a specific action	Action Specific

Printing out Results

Reusing Action Results

action-results.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Playbook
metadata:
  name: use-previous-action-result
spec:
  description: Creates a file with the content of the config
  configs:
    - types:
        - Kubernetes::Pod
  actions:
    - name: Fetch all changes
      sql:
        query: SELECT id FROM config_changes WHERE config_id = '{{.config.id}}'
        driver: postgres
        connection: connection://postgres/local
    - name: Send notification
      if: 'last_result().count > 0'
      notification:
        title: 'Changes summary for {{.config.name}}'
        connection: connection://slack/flanksource
        message: |
          {{$rows:=index last_result "count"}}
          Found {{$rows}} changes

Logs​

Post processing​

Time range​

CloudWatch​

Kubernetes​

Loki​

OpenSearch​

Output​

Log line​

Using Results in Subsequent Actions​

Templating​

CEL Expressions​

Go Templating​

Functions​

Logs

Post processing

Time range

CloudWatch

Kubernetes

Loki

OpenSearch

Output

Log line

Using Results in Subsequent Actions

Templating

CEL Expressions

Go Templating

Functions