Notifications

Mission Control provides a flexible, event-based notification system that enables filtering and templating of notifications via email, push, slack, teams, etc.

Notifications are triggered by events. An event is a specific occurrence or change in state within the system. For example, notifications can be granular such as a health check failing or a pod crashlooping - or high level based on the health of the system as a whole.

check-failed.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
  name: http-check
  namespace: default
spec:
  events:
    - check.failed
  to:
    email: alerts@acme.com

This notification sends an email to alerts@acme.com whenever a health check fails.

Field	Description	Scheme	Template Env
events*	A list of events that should trigger this notification
filter	Filter narrows down the event trigger. Example: check.type == 'http' filter on check.failed event will only send notification for failing http checks.	CEL	CheckEvents ConfigEvents ComponentEvents
repeatGroup	RepeatGroup allows notifications to be grouped by certain set of keys and only send one per group within the specified repeat interval. Valid group keys: resource_id & source_event.
repeatInterval	The waiting time to resend a notification after it has been succefully sent.	Duration
template	Channel dependent e.g. email body for email	Go Template	CheckEvents ConfigEvents ComponentEvents
title	Channel dependent e.g. subject for email	Go Template	CheckEvents ConfigEvents ComponentEvents
to.connection	Connection to external service	Connection
to.email	Send an email to any email address
to.person	Send an email to the person
to.playbook	specify the <namespace>/<name> of the playbook that should be triggered when this notification fires.	string
to.properties	Properties are channel dependent special directives to modify the notification message. Example: for email, FromAddress=admin@flanksource.com modifies the sender of the email. Read more
to.team	Send an email to every person in the team
to.url	Custom notification URL
waitFor	The duration to delay sending a health-based notification. After this period, the health status is reassessed to confirm it hasn't changed, helping prevent false alarms from transient issues.	Duration
waitForEvalPeriod	an additional delay after WaitFor before evaluating Kubernetes config health	Duration

Single Recipient

Only one recipient can be specified

Templating

You can use Go Templates to customize the content of notification titles and bodies. This allows you to include dynamic information from the event that triggered the notification.

For example, the following notification definition uses a Go Template to include the name and health status of the check in the notification title:

notification-template
apiVersion: mission-control.flanksource.com/v1
kind: Notification
spec:
  events:
    - check.failed
    - check.passed
  title: Check {{.check.name}} is {{.check.health}}

In this example {{.check.name}} and {{.check.health}} are placeholders that gets replaced with the actual values of the name and health fields from the check object associated with the event.

Markdown

Mission Control defines all notification templates in markdown and converts them to HTML or plain text based on the notification channel's support

Filtering Events

Mission Control allows you to fine-tune your notification delivery by filtering events using CEL. This enables you to specify precise conditions that dictate when a notification should be triggered. For instance, you can configure notifications to be sent only for specific event types, specific resources, or when certain conditions are met.

The CEL filters you define have access to the same set of variables as the templates, providing flexibility in defining your filtering criteria. These variables, representing various attributes and properties of the events, empower you to create highly targeted notifications.

To illustrate, consider a scenario where you want to receive notifications exclusively for failed HTTP checks. The following example demonstrates how to achieve this using a CEL filter:

filter-http.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
  name: http-alerts
spec:
  events:
    - check.failed
  # Filter for events where the check type is 'http'
  filter: check.type == 'http'
  to:
    email: alerts@acme.com

In this example:

• The filter field specifies the CEL expression check.type == 'http'.
• This expression evaluates to true only when the type attribute of the check object is equal to 'http'.
• So, Mission Control only sends notifications for events that meet this condition, which alerts you when HTTP checks fail.

Triggering Playbooks

Playbooks can be configured as recipients for notifications, allowing you to trigger automated workflows instead of sending notifications to traditional channels like email or Slack.

While playbooks themselves are capable of listening to events like notifications - this approach leverages the full capabilities of notifications, including:

• Rate Limitation
• Silences
• Repeat Intervals
• WaitFor

playbook-on-unhealthy-configs.yaml
---
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
  name: config-healths
spec:
  events:
    - config.healthy
    - config.unhealthy
    - config.warning
  to:
    playbook: mc/echo-config

This notification above triggers the playbook mc/echo-config whenever a config changes its health to the specified list.

Permissions

To enable a notification to execute a playbook, the notification must have playbook:run permission on the playbook.

The example shows two notifications: check-alerts and homelab-config-health-alerts which belong to a permission group "config-notifications". The group has playbook:run permission, which both notifications inherit.

permission.yaml
---
# yaml-language-server: $schema=../../config/schemas/permission.schema.json
apiVersion: mission-control.flanksource.com/v1
kind: Permission
metadata:
  name: allow-config-notifications-to-run-playbook
spec:
  description: allow config notifications to run playbook
  subject:
    notification: config-notifications
  actions:
    - playbook:run
    - playbook:approve
  object:
    playbooks:
      - name: echo-config
---
# yaml-language-server: $schema=../../config/schemas/permissiongroup.schema.json
apiVersion: mission-control.flanksource.com/v1
kind: PermissionGroup
metadata:
  name: config-notifications
spec:
  name: config-notifications
  notifications:
    - name: check-alerts
      namespace: mc
    - name: homelab-config-health-alerts
      namespace: mc