Alertmanager

Fetches alerts from the Prometheus Alertmanager /api/v2/alerts endpoint and stores the alerts in results.alerts.

The following example transforms results.alerts so that each alert becomes a single check result. To fail when Alertmanager returns alerts, add a test expression in an imported fixture, such as results.alerts.size() == 0.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: http://kube-prometheus-stack-alertmanager.monitoring:9093
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
alerts	Alert name regular expressions to include	[]string
connection	Connection reference or URL to use. Inline username and password can supply credentials when the check supports them	Connection
exclude_filters	Label values to exclude	[map[string]string]
filters	Label regular expressions to include	[map[string]string]
ignore	Alert name regular expressions to exclude	[]string
password		EnvVar
url	Prometheus Alertmanager URL. Required unless connection supplies the URL	string
username		EnvVar
dependsOn	Checks that must complete before this check runs	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	[map[string]string]
markFailOnEmpty	If a transformation or datasource returns empty results, the check should fail	boolean
metrics	Metrics to export from	[]Metrics
namespace	Namespace to insert the check result into when it differs from the canary namespace	string
relationships	Relationships that link check results to components or configs	Relationship
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression
transformDeleteStrategy	Status to apply to transformed checks when the source check no longer returns them	string

Relationships

Field	Description	Scheme
components	Select components to link to this check	Lookup
configs	Select configs to link to this check	Lookup

Lookup

Field	Description	Scheme
name	Name selector for the related resource	Lookup Selector
namespace	Namespace selector for the related resource	Lookup Selector
type	Type selector for the related resource	Lookup Selector

Lookup selector

Lookup specifies the type of lookup to perform.

Field	Description	Scheme
expr	Expr is a cel-expression	string
label	Label specifies the key to lookup on the label	string
value	Value is the static value to use	string

Inserting checks into different namespaces

You can specify different namespaces for checks using the namespace field. This is helpful when checks are dynamically generated via transformation and need to be assigned their respective namespace.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: http://kube-prometheus-stack-alertmanager.monitoring:9093
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

The Alertmanager canary can reside in any namespace. The transformed checks use the namespace derived from the alerts.