Prometheus

The Prometheus Check connects to the Prometheus host, performs the desired query, and displays the results.

prometheus.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus
spec:
  schedule: "@every 5m"
  prometheus:
    - url: http://kube-prometheus-stack-prometheus.monitoring:9090
      name: prometheus-check
      query: kubernetes_build_info{job!~"kube-dns|coredns"}
      display:
        expr: results[0].git_version

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
query*	PromQL query to execute
url*	Prometheus instance, defaults to --prometheus command line argument
bearer	Bearer token to use for authentication	EnvVar
oauth		OAuth
password	Password for basic auth	EnvVar
tls	TLS and mutual TLS configuration	TLSConfig
username	Username for basic auth	EnvVar
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	[map[string]string]
markFailOnEmpty	If a transformation or datasource returns empty results, the check should fail	boolean
metrics	Metrics to export from	[]Metrics
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression

TLS Config

Use tls.ca to verify Prometheus with a custom CA. Add tls.cert and tls.key for mutual TLS.

Field	Description	Scheme
ca	PEM encoded CA certificate to verify the Prometheus server certificate	EnvVar
cert	PEM encoded client certificate	EnvVar
handshakeTimeout	TLS handshake timeout. Defaults to 10 seconds	Duration
insecureSkipVerify	Skip verification of the server certificate chain and host name	boolean
key	PEM encoded client private key	EnvVar

OAuth

Field	Description	Scheme
clientID	Client ID	EnvVar
clientSecret	Client Secret	EnvVar
params	OAuth parameters when requesting a token	map[string]string
scope	Scopes to request	[]string
tokenURL	Token URL	string

Result Variables

Name	Description	Scheme
value		float
firstValue	Number of rows returned	int
results	A list of results	[]map[string]interface

Examples

Create a check per prometheus job using a transform

jobs.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"}
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': r["value"] > 0
          }).toJSON()

Create a check per failing job only

jobs.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-failing-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"} == 0
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': false
          }).toJSON()

Prometheus with mutual TLS

prometheus-mtls.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-mtls
spec:
  schedule: "@every 1m"
  prometheus:
    - name: Prometheus mutual TLS
      url: https://prometheus.example.com
      query: up
      tls:
        ca:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: ca.crt
        cert:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: client.crt
        key:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: client.key