Kubernetes

The Kubernetes check queries Kubernetes resources such as Pods and evaluates their health or readiness.

kubernetes.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-system-checks
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: kube-system
      kind: Pod
      healthy: true
      # resource:
      #  search: labels.app=test
      #      OR
      #  labelSelector: k8s-app=kube-dns
      namespaceSelector:
        name: kube-*,!*lease
      # name: "*"
      display:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, !k8s.isHealthy(i)).
          map(i, "%s/%s -> %s".format([i.metadata.namespace, i.metadata.name, k8s.getHealth(i).message])).join('\n')
      test:
        expr: dyn(results).all(x, k8s.isHealthy(x))

Field	Description	Scheme
kind*	Kubernetes object kind	string
name*	Name of the check, must be unique within the canary	string
apiVersion	Kubernetes API version for the resource	string
cnrm	CNRM connection details	CNRM
connection	The connection url to use, mutually exclusive with kubeconfig	Connection
eks	EKS connection details	EKS
gke	GKE connection details	GKE
healthy	Fail the check if any resources are unhealthy	boolean
ignore	Exclude resources by name. Values are glob patterns.	[]string
kubeconfig	Source for kubeconfig	EnvVar
namespaceSelector	Filters namespaces by name or labels	Resource Selector
ready	Fail the check if any resources are not ready	boolean
resource	Filters resources by name, namespace, or labels	Resource Selector
dependsOn	Checks that must complete before this check runs	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	[map[string]string]
markFailOnEmpty	If a transformation or datasource returns empty results, the check should fail	boolean
metrics	Metrics to export from	[]Metrics
namespace	Namespace to insert the check result into when it differs from the canary namespace	string
relationships	Relationships that link check results to components or configs	Relationship
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression
transformDeleteStrategy	Status to apply to transformed checks when the source check no longer returns them	string

Resource Selector

Field	Description	Scheme
fieldSelector	Kubernetes field selector	string
labelSelector	Kubernetes label selector	string
name	Resource name or glob pattern	string
search	Search expression	string

The following example uses namespaceSelector and resource to select Kubernetes resources.

kubernetes.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-pass
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: k8s-ready pods
      kind: Pod
      namespaceSelector:
        name: canaries
      resource:
        name: k8s-check-ready
    - name: k8s-ready pods

      kind: Pod
      namespaceSelector:
        name: canaries
      ready: false
      resource:
        # resource name supports wildcards
        name: k8s-check-not-*
        #labelSelector: app=k8s-not-ready

API Version

Use apiVersion when the same Kubernetes kind exists in more than one API group.

kubernetes-apiversion.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-apiversion-pass
spec:
  schedule: "@every 5m"
  kubernetes:
    # Same Kind (Event), different apiVersions
    - name: events-core
      kind: Event
      apiVersion: v1
      namespaceSelector:
        name: kube-system

    - name: events-k8s-io
      kind: Event
      apiVersion: events.k8s.io/v1
      namespaceSelector:
        name: kube-system

Healthy

Using healthy: true is functionally equivalent to this CEL expression:

dyn(results).all(x, k8s.isHealthy(x))

The following example uses healthy: true.

kubernetes-healthy.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-system-checks
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: kube-system
      kind: Pod
      healthy: true
      # resource:
      #  search: labels.app=test
      #      OR
      #  labelSelector: k8s-app=kube-dns
      namespaceSelector:
        name: kube-*,!*lease
      # name: "*"
      display:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, !k8s.isHealthy(i)).
          map(i, "%s/%s -> %s".format([i.metadata.namespace, i.metadata.name, k8s.getHealth(i).message])).join('\n')
      test:
        expr: dyn(results).all(x, k8s.isHealthy(x))

See the CEL function k8s.isHealthy for more details

Ready

Similar to the healthy flag, the ready flag is functionally equivalent to this CEL expression:

dyn(results).all(x, k8s.isReady(x))

---

Checking for certificate readiness

cert-manager.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: cert-manager
spec:
  schedule: "@every 15m"
  kubernetes:
    - name: cert-manager-check
      kind: Certificate
      test:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, i.status.conditions[0].status != "True").size() == 0
      display:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, i.status.conditions[0].status != "True").
          map(i, "%s/%s -> %s".format([i.metadata.namespace, i.metadata.name, i.status.conditions[0].message])).join('\n')

Remote clusters

A single canary-checker instance can connect to any number of remote clusters via custom kubeconfig. Either the kubeconfig itself or the path to the kubeconfig can be provided.

kubeconfig from kubernetes secret

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      description: "deploy httpbin"
      kubeconfig:
        valueFrom:
          secretKeyRef:
            name: aws-kubeconfig
            key: kubeconfig
      kind: Pod
      ready: true
      namespaceSelector:
        name: default

Kubeconfig inline

remote-cluster.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      kubeconfig:
        value: |
          apiVersion: v1
          clusters:
              - cluster:
                  certificate-authority-data: xxxxx
                  server: https://xxxxx.sk1.eu-west-1.eks.amazonaws.com
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          contexts:
              - context:
                  cluster: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                  namespace: mission-control
                  user: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          current-context: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          kind: Config
          preferences: {}
          users:
              - name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                user:
                  exec:
                      ....
      kind: Pod
      ready: true
      namespaceSelector:
        name: default

Kubeconfig from local filesystem

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      kubeconfig:
        value: /root/.kube/aws-kubeconfig
      kind: Pod
      ready: true
      namespaceSelector:
        name: default