Transformation

Transformations modify scraped configuration items before they are saved to the database. They are applied in the following order:

1. Script transformations - Modify config using CEL, Go template, JSONPath, or JavaScript
2. Field exclusions - Remove specified fields from the config
3. Masking - Replace sensitive fields with hashes or static strings
4. Relationships - Create links between configuration items
5. Change transformations - Filter and categorize detected changes
6. Locations & Aliases - Add metadata for grouping and lookup

Transform

Field	Description	Scheme
aliases	Add config aliases for lookup	[]LocationOrAlias
changes.exclude	CEL expressions to ignore changes	[]CEL
changes.mapping	Categorize and modify changes	[]Mapping
exclude	Remove fields from config	[]Exclude
expr	CEL expression to transform config	CEL
gotemplate	Go template expression	Go Template
javascript	JavaScript expression	JavaScript
jsonpath	JSONPath expression	JSONPath
locations	Add location metadata for grouping	[]LocationOrAlias
mask	Replace sensitive fields with hash or static string	[]Mask
relationship	Create relationships between config items	[]Relationship

Script Context

When using script transformations (expr, gotemplate, jsonpath, javascript), these variables are available:

Field	Description	Scheme
config	Raw configuration data as JSON	JSON
last_scrape_summary	Summary from the previous scrape run. Contains per-type counts (config_types map with added/updated/unchanged/changes/deduped) and entity counts (external_users, external_groups, external_roles, config_access, access_logs each with scraped/saved/skipped/deleted).	ScrapeSummary
result	Current scrape result with metadata (id, name, config_type, labels, tags, etc.)	ScrapeResult

Script transformations must return a JSON array of ScrapeResult objects.

Scrape Summary

The last_scrape_summary provides counts from the previous scrape run, useful for conditional logic (e.g. skip processing if no changes).

Field	Description	Scheme
access_logs	Entity counts for access log entries	EntitySummary
config_access	Entity counts for config access records	EntitySummary
config_types	Map of config type name to per-type counts	map[string]ConfigTypeSummary
external_groups	Entity counts for external groups	EntitySummary
external_roles	Entity counts for external roles	EntitySummary
external_users	Entity counts for external users	EntitySummary

Where ConfigTypeSummary has fields: added, updated, unchanged, changes, deduped (all integers). And EntitySummary has fields: scraped, saved, skipped, deleted (all integers).

cel-transform-example.yaml
transform:
  expr: |
    [result.merge({
      'config': config.merge({
        'normalized_name': config.name.lowerAscii().replace(' ', '-'),
        'environment': config.tags.?env.orValue('unknown')
      })
    })].toJSON()

Field Exclusions

Remove fields from the scraped configuration. Useful for removing sensitive data or fields that change frequently without material impact.

Field	Description	Scheme
jsonpath*	JSONPath to the field(s) to remove	JSONPath
types	Config types to apply exclusion to. If empty, applies to all types.	[]string

exclude-managed-fields.yaml
transform:
  exclude:
    - types:
        - Kubernetes::Pod
      jsonpath: '.metadata.managedFields'
    - types:
        - Kubernetes::Node
      jsonpath: '.status.images'

Masking

Replace sensitive fields with a hash or static string. Using a hash enables change detection without exposing the actual value.

Field	Description	Scheme
jsonpath*	JSONPath to the field(s) to mask	JSONPath
value*	Hash function (md5sum) or static replacement string	string
selector	CEL expression to select which configs to mask	CEL

Hash Functions

Function	Description
md5sum	MD5 hash of the value - enables change detection while masking content

mask-secrets.yaml
transform:
  mask:
    - selector: config_type == "Kubernetes::Secret"
      jsonpath: $.data
      value: md5sum
    - selector: config_type.startsWith("AWS::")
      jsonpath: $.Credentials
      value: '***REDACTED***'

Relationships

Create relationships between configuration items. See Relationships Guide for detailed usage.

Field	Description	Scheme
filter*	CEL expression to select which configs to apply the relationship to	CEL
agent	Agent of the related config (self for local)	Lookup
expr	CEL expression returning a list of relationship selectors (for dynamic relationships)	CEL
external_id	External ID of the related config	Lookup
id	ID or external ID of the related config	Lookup
labels	Labels to match on the related config	map[string]Lookup
name	Name of the related config	Lookup
namespace	Namespace of the related config	Lookup
parent	If true, matched configs become parents; if false, they become children	boolean
scope	Scope for the related config (e.g. scraper ID). Use all to disregard scope.	Lookup
type	Type of the related config	Lookup

Lookup

Lookup provides multiple ways to specify a value for relationship matching. At least one of expr, value, or label must be specified.

Field	Description	Scheme
expr	CEL expression returning the value	CEL
label	Get value from an existing label	string
value	Static value	string

relationship-static.yaml
transform:
  relationship:
    - filter: config_type == "Kubernetes::Service"
      type:
        value: 'Kubernetes::Deployment'
      name:
        expr: |
          has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : ''

Dynamic Relationships

For complex relationship logic, use an expr that returns a list of ResourceSelectors:

relationship-dynamic.yaml
transform:
  relationship:
    - filter: config_type == 'Kubernetes::Pod'
      expr: |
        config.spec.volumes.
          filter(item, has(item.persistentVolumeClaim)).
          map(item, {
            "type": "Kubernetes::PersistentVolumeClaim",
            "name": item.persistentVolumeClaim.claimName
          }).
          toJSON()

Change Transformations

Transform detected changes by excluding irrelevant ones or categorizing them.

Change Exclusions

CEL expressions that exclude changes from being recorded. Uses Change Context.

exclude-node-image-changes.yaml
transform:
  changes:
    exclude:
      - 'config_type == "Kubernetes::Node" && details.message == "status.images"'

Change Mapping

Field	Description	Scheme
filter*	CEL expression to select which changes to apply mapping to	CEL
action	delete marks the config as deleted, ignore skips the change, move-up redirects to an ancestor, copy-up copies to an ancestor, move redirects to a target, copy copies to targets	delete | ignore | move-up | copy-up | move | copy
ancestor_type	Config type of ancestor to target for move-up/copy-up. Walks the parent chain until a match is found. If omitted, the immediate parent is used.	string
config_id	CEL expression returning the target config's external ID for redirecting changes.	CEL
config_type	Target config type for redirecting changes.	string
scraper_id	Scraper ID for target config lookup. Use "all" for cross-scraper lookups.	string
severity	Set the change severity	string
summary	Replace the change summary	Go Template
target	Selector for copy/move actions. Mutually exclusive with ancestor_type.	RelationshipSelectorTemplate
type	Set the change type	string

categorize-changes.yaml
transform:
  changes:
    mapping:
      - filter: >
          change.change_type == 'diff' &&
          change.summary == "status.containerStatuses" &&
          patch != null && has(patch.status) &&
          has(patch.status.containerStatuses) &&
          patch.status.containerStatuses.size() > 0 &&
          has(patch.status.containerStatuses[0].restartCount)
        type: PodCrashLooping
      - filter: >
          change.change_type == 'diff' &&
          change.summary == "status.images" &&
          config.kind == "Node"
        type: ImageUpdated

move-change-to-parent-cluster.yaml
transform:
  changes:
    mapping:
      # Move pod crash events up to the Cluster
      - filter: change.change_type == "PodCrashLooping"
        action: move-up
        ancestor_type: Kubernetes::Cluster
      # Copy deployment changes to related service configs
      - filter: change.change_type == "diff" && config.config_type == "Kubernetes::Deployment"
        action: copy
        target:
          type: Kubernetes::Service
          labels:
            app: '$(.config.labels.app)'

Locations

Add location metadata to config items for grouping and filtering.

Field	Description	Scheme
type*	Config types to apply to (supports wildcards like AWS::*)	MatchExpression
filter	CEL expression that must return true for the location to apply	CEL
values	Location values (supports Go templates)	[]string
withParent	Parent type for hierarchical locations	string

add-locations.yaml
transform:
  locations:
    - type: "AWS::*"
      values:
        - "{{.config.Region}}"
    - type: "Kubernetes::*"
      filter: has(config.metadata.namespace)
      values:
        - "{{.config.metadata.namespace}}"

Aliases

Add alternative identifiers for config items to enable lookup by different names.

Field	Description	Scheme
type*	Config types to apply to (supports wildcards like AWS::*)	MatchExpression
filter	CEL expression that must return true for the alias to apply	CEL
values	Alias values (supports Go templates)	[]string
withParent	Parent type for scoped aliases	string

add-aliases.yaml
transform:
  aliases:
    - type: "Kubernetes::Pod"
      filter: has(config.metadata.labels.app)
      values:
        - "{{.config.metadata.labels.app}}"

Complete Example

comprehensive-transform.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: production-scraper
spec:
  kubernetes:
    - clusterName: production
      transform:
        # Script transformation - normalize labels
        expr: |
          [result.merge({
            'labels': result.labels.filter(k, !k.startsWith('internal.'))
          })].toJSON()

        # Remove verbose fields
        exclude:
          - types:
              - Kubernetes::Pod
            jsonpath: '.metadata.managedFields'
          - types:
              - Kubernetes::Node
            jsonpath: '.status.images'

        # Mask sensitive data
        mask:
          - selector: config_type == "Kubernetes::Secret"
            jsonpath: $.data
            value: md5sum

        # Build relationships
        relationship:
          - filter: config_type == "Kubernetes::Service"
            type:
              value: 'Kubernetes::Deployment'
            name:
              expr: |
                has(config.spec.selector.app) ? config.spec.selector.app : ''

        # Categorize changes
        changes:
          mapping:
            - filter: change.summary == "status.images"
              type: ImageUpdate
          exclude:
            - 'config_type == "Kubernetes::Node" && change.summary == "status.conditions"'

        # Add locations
        locations:
          - type: "Kubernetes::*"
            filter: has(config.metadata.namespace)
            values:
              - "{{.config.metadata.namespace}}"

        # Add aliases
        aliases:
          - type: "Kubernetes::Pod"
            filter: has(config.metadata.labels.app)
            values:
              - "{{.config.metadata.labels.app}}"

Use Cases

Compliance - Masking Sensitive Data

transform:
  mask:
    - selector: config_type == "Kubernetes::Secret"
      jsonpath: $.data
      value: md5sum
    - selector: config_type == "AWS::IAM::User"
      jsonpath: $.AccessKeys[*].SecretAccessKey
      value: '***'

Change Management - Categorizing Changes

transform:
  changes:
    mapping:
      - filter: change.change_type == 'diff' && change.summary.contains('restart')
        type: Restart
        severity: medium
      - filter: change.change_type == 'diff' && change.summary.contains('scale')
        type: ScaleEvent

Topology - Building Relationships

transform:
  relationship:
    - filter: config_type == "AWS::EC2::Instance"
      expr: |
        config.SecurityGroups.map(sg, {
          'type': 'AWS::EC2::SecurityGroup',
          'external_id': sg.GroupId
        }).toJSON()

Data Normalization

transform:
  expr: |
    [result.merge({
      'config': config.merge({
        'environment': config.tags.?Environment.orValue(
          config.tags.?env.orValue('unknown')
        ).lowerAscii()
      })
    })].toJSON()