Prometheus

The Prometheus check connects to Prometheus, runs a PromQL query, and displays the results.

prometheus.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus
spec:
  schedule: "@every 5m"
  prometheus:
    - url: http://kube-prometheus-stack-prometheus.monitoring:9090
      name: prometheus-check
      query: kubernetes_build_info{job!~"kube-dns|coredns"}
      display:
        expr: results[0].git_version

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
query*	PromQL query to execute
awsSigV4	AWS Signature Version 4 signing configuration	AWSSigV4
bearer	Bearer token to use for authentication	EnvVar
connection	Connection URL that supplies Prometheus connection details	string
headers	HTTP headers to send with the query request	[]EnvVar
host	Deprecated. Use url; setting host fails the check.	string
oauth		OAuth
password	Password for basic auth	EnvVar
tls	TLS and mutual TLS configuration	TLSConfig
url	Prometheus URL. Required unless connection or --prometheus supplies the URL	string
username	Username for basic auth	EnvVar
dependsOn	Checks that must complete before this check runs	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	[map[string]string]
markFailOnEmpty	If a transformation or datasource returns empty results, the check should fail	boolean
metrics	Metrics to export from	[]Metrics
namespace	Namespace to insert the check result into when it differs from the canary namespace	string
relationships	Relationships that link check results to components or configs	Relationship
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression
transformDeleteStrategy	Status to apply to transformed checks when the source check no longer returns them	string

TLS Config

Use tls.ca to verify Prometheus with a custom CA. Add tls.cert and tls.key for mutual TLS.

Field	Description	Scheme
ca	PEM encoded CA certificate to verify the Prometheus server certificate	EnvVar
cert	PEM encoded client certificate	EnvVar
handshakeTimeout	TLS handshake timeout. Defaults to 10 seconds	Duration
insecureSkipVerify	Skip verification of the server certificate chain and host name	boolean
key	PEM encoded client private key	EnvVar

OAuth

Field	Description	Scheme
clientID	Client ID	EnvVar
clientSecret	Client Secret	EnvVar
params	OAuth parameters when requesting a token	map[string]string
scope	Scopes to request	[]string
tokenURL	Token URL	string

Result Variables

Name	Description	Scheme
value	First sample value, or 0 when the query returns no samples	float
firstResult	First result map, including labels and value	map[string]interface
results	List of result maps. Each map contains labels and value.	[]map[string]interface

Examples

Create a check per Prometheus job using a transform

jobs.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"}
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': r["value"] > 0
          }).toJSON()

Create a check per failing job only

jobs-fail-only.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-failing-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"} == 0
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': false
          }).toJSON()

Prometheus with mutual TLS

prometheus-mtls.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-mtls
spec:
  schedule: "@every 1m"
  prometheus:
    - name: Prometheus mutual TLS
      url: https://prometheus.example.com
      query: up
      tls:
        ca:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: ca.crt
        cert:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: client.crt
        key:
          valueFrom:
            secretKeyRef:
              name: prometheus-mtls
              key: client.key