Folder

The Folder check verifies folder contents by size, age, and count. Folder-based checks are useful for:

• Verifying that backups have been uploaded and are the appropriate size
• Checking that logs or other temporary files are being cleaned up
• For batch processes:
  • Checking whether files are being processed or produced
  • Checking the size of a queue processing backlog
  • Checking whether error files, such as .err or .log, have been produced

folder-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: folder-check
spec:
  schedule: "@every 5m"
  folder:
    - path: /etc/
      name: folder-check-min
      description: Checks if there are at least 10 files in the folder
      minCount: 10

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
path*	Local path or remote folder path. Supported remote paths are smb://, UNC paths, s3://, and gcs://. For SFTP, set sftpConnection and use the remote path.
availableSize	Minimum available size on the filesystem	Size
awsConnection	AWS Access credentials	S3
filter.maxAge	Include files modified within this age	Duration
filter.maxSize	Include files no larger than this size	Size
filter.minAge	Include files older than this age	Duration
filter.minSize	Include files at least this size	Size
filter.regex	Filter files based on regular expression	regex
filter.since	Include files modified since this time	string
gcpConnection	Connection details for GCP	GCP
maxAge	The oldest age a file can be, often used to check for unprocessed files or files that have not been cleaned up	Duration
maxCount	The maximum number of files inside the path, can be used in conjunction with filter.regex to detect error files	integer
maxSize	The maximum file size	Size
minAge	The youngest age a file can be	Duration
minCount	The minimum number of files inside the path	integer
minSize	The minimum file size, can be used to detect backups that did not upload successfully	Size
recursive	When set to true, recursively scans the folder to list all files (symlinks are listed but not traversed)	boolean
sftpConnection	SFTP connection details. Use this field with a remote path.	SFTP
smbConnection	SMB connection details	SMB
totalSize	Minimum total size of the filesystem	Size
dependsOn	Checks that must complete before this check runs	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	[map[string]string]
markFailOnEmpty	If a transformation or datasource returns empty results, the check should fail	boolean
metrics	Metrics to export from	[]Metrics
namespace	Namespace to insert the check result into when it differs from the canary namespace	string
relationships	Relationships that link check results to components or configs	Relationship
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression
transformDeleteStrategy	Status to apply to transformed checks when the source check no longer returns them	string

The following example checks a local folder for files matching pg-backups-.*.zip. The check fails if no matching files were modified within the last 1d, or if the matching files are smaller than 10mb.

postgres-backup-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: folder-check
spec:
  schedule: "@every 5m"
  folder:
    - name: pg-backup checks
      path: /data/backups
      filter:
        regex: "pg-backups-.*.zip"
      maxAge: 1d # require a daily backup
      minSize: 10mb # the backup should be at least 10mb

Result Variables

The following fields are available in test, display, and transform expressions.

Field	Scheme
oldest	File
newest	File
smallest	File
largest	File
size	int64
availableSize	int64
files	[]File

File contains name, size, mode, modified, and is_dir.

Connection Types

SFTP

Field	Description	Scheme
connection	The connection url to use, mutually exclusive with host and port	Connection
host	SFTP host	string
password		EnvVar
port	SFTP port	integer
username		EnvVar

SFTP Check

exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: sftp-pass
spec:
  schedule: "@every 5m"
  folder:
    - path: /tmp
      name: sample sftp check
      sftpConnection:
        host: 192.168.1.5
        username:
          value: <username>
        password:
          value: <password>
      maxCount: 10

S3

Field	Description	Scheme
accessKey	Access Key ID	EnvVar
assumeRole	Role ARN to assume	string
connection	The connection url to use, mutually exclusive with accessKey and secretKey	Connection
endpoint	Custom AWS Endpoint to use	string
objectPath	Object key prefix for S3 requests. The bucket comes from path, such as s3://bucket-name.	string
region	The AWS region	string
secretKey	Secret Access Key	EnvVar
sessionToken	Session token	EnvVar
skipTLSVerify	Skip TLS verify when connecting to AWS	boolean
usePathStyle	Use path-style requests for S3-compatible endpoints.	boolean

S3 Bucket

exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: s3-bucket-pass
  namespace: canaries
spec:
  schedule: "@every 5m"
  folder:
    # Check for any backup not older than 7 days and min size 25 bytes
    - path: s3://tests-e2e-1
      name: mysql backup check
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "(.*)backup.zip$"
      maxAge: 7d
      minSize: 25b
    # Check for any mysql backup not older than 7 days and min size 25 bytes
    - path: s3://tests-e2e-1
      name: mysql backup check (7 days and 25 bytes size)
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "mysql\\/backups\\/(.*)\\/mysql.zip$"
      maxAge: 7d
      minSize: 25b
    # Check for any pg backup not older than 7 days and min size 50 bytes
    - path: s3://tests-e2e-1
      name: pg backup check (7 days and 50 bytes size)
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "pg\\/backups\\/(.*)\\/backup.zip$"
      maxAge: 7d
      minSize: 25b
    - name: recursive folders
      namespace: default
      path: s3://recursive-test/developers/**/*.txt
      minCount: 4
      display:
        expr: results.?files.orValue([]).map(i, i.name).join(", ")
      awsConnection:
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY

SMB

Field	Description	Scheme
connection	The connection url to use, mutually exclusive with host, share, and credentials	Connection
domain	Windows Domain
host	The SMB host	string
password		EnvVar
port		integer
username		EnvVar

SMB / CIFS

smb.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: smb-pass
spec:
  schedule: "@every 5m"
  folder:
    # Check for any backup not older than 7 days and min size 25 bytes
    - path: \\windows-server\sharename\folder
      smbConnection:
        username:
          valueFrom:
            secretKeyRef:
              name: smb-credentials
              key: USERNAME
        password:
          valueFrom:
            secretKeyRef:
              name: ssmb-credentials
              key: PASSWORD
      filter:
        regex: "(.*)backup.zip$"
      maxAge: 7d
      minSize: 25b

GCS

Field	Description	Scheme
connection	The connection url to use, mutually exclusive with credentials	Connection
credentials	The credentials to use for authentication	EnvVar
endpoint	Custom GCP Endpoint to use	string
skipTLSVerify	Skip TLS verification when connecting to GCP	boolean

GCS Check

gcs.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: recursive-folder-check
spec:
  schedule: "@every 5m"
  folder:
    - path: gcs://folder-check-test/recursive-test
      name: recursive folders
      namespace: default
      minCount: 3
      recursive: true
      display:
        expr: results.?files.orValue([]).map(i, i.name).join(", ")
      gcpConnection:
        connection: connection://gcs/flanksource-prod