Playwright

The playwright scraper runs a Playwright script in a headless browser and scrapes its output as configuration items. Use it for systems that expose no usable API — cloud consoles, admin portals, or SaaS dashboards — where the only reliable source of configuration is the web UI.

The script writes JSON to stdout, which is scraped exactly as the exec scraper does. It can also record HAR files, videos and screenshots as artifacts.

playwright-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: playwright-simple
spec:
  schedule: "@every 1h"
  playwright:
    - type: Playwright::Page
      id: $.url
      name: $.title
      har: true
      script: |
        #!/usr/bin/env node
        const { chromium } = require('playwright');
        (async () => {
          const browser = await chromium.launch();
          const contextOpts = {};
          if (process.env.PW_HAR_PATH) {
            contextOpts.recordHar = { path: process.env.PW_HAR_PATH, mode: 'full' };
          }
          const context = await browser.newContext(contextOpts);
          const page = await context.newPage();
          await page.goto('https://www.google.com/finance/');
          await page.waitForLoadState('networkidle');
          const title = await page.title();
          const url = page.url();
          await context.close();
          await browser.close();
          console.log(JSON.stringify({ url, title }));
        })();

Scraper

Field	Description	Scheme
`schedule`	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	Cron
`full`	Set to `true` to extract changes and access logs from scraped configurations. Defaults to `false`.	`bool`
`retention`	Settings for retaining changes, analysis and scraped items	`Retention`
`playwright`	Specifies the list of Playwright configurations to scrape.	`[]Playwright`
`logLevel`	Specify the level of logging.	`string`

Playwright

Field	Description	Scheme
`script*`	Inline TypeScript/JavaScript to run with Playwright. The script should output JSON to stdout.	`string`
`artifacts`	Additional artifact paths to collect after execution	[]Artifact
`checkout`	Checkout a git repository before running the script	GitConnection
`connections`	Connections for AWS/GCP/Azure/Kubernetes credential injection	ExecConnections
`env`	Environment variables to set during execution	[]EnvVar
`har`	Record a HAR (HTTP Archive) file. Shorthand for `trace.har`	`boolean`
`headless`	Run the browser headless. Defaults to `true`	`boolean`
`login`	Log in automatically before the script runs	Login
`outputMode`	How stdout is parsed. Defaults to `json`	`json` \| `raw`
`query`	Export existing config items as JSON files that the script can read	[]Query
`timeout`	Timeout in seconds for the script execution. Defaults to `300`	`integer`
`trace`	Configure HAR, video and network recording	Trace
`labels`	Labels for each config item.	`map[string]string`
`properties`	Custom templatable properties for the scraped config items.	`[]ConfigProperty`
`tags`	Tags for each config item. Max allowed: 5	`[]ConfigTag`
`transform`	Transform configs after they've been scraped	`Transform`

Trace

Field	Description	Scheme
`har`	Record a HAR file of the session	`bool`
`domains`	Domain patterns to include in the recording. Prefix a pattern with `!` to exclude it — useful for stripping out telemetry and widget traffic	`[]string`
`video`	When to keep a video recording e.g. `on-error`, `always`	`string`

Specify one login provider.

Field	Description	Scheme
`aws`	Log in to the AWS console via federation	`AWS`
`browser`	Reuse cookies from an existing browser connection	`Browser`

Also accepts every field of an AWSConnection.

Field	Description	Scheme
`login`	Name of the federation login to use	`string`
`issuer`	Issuer passed to the AWS federation endpoint	`string`
`sessionDuration`	Length of the federated session, in seconds	`int`

Field	Description	Scheme
`connection`	Connection holding the browser cookies to reuse	`string`

Example: AWS console screenshots

playwright-aws-rds.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: aws-rds-console-screenshots
  annotations:
    playwright.keep: "true"
spec:
  schedule: "@every 6h"
  playwright:
    - type: AWS::RDS::DBInstance
      id: $.instanceId
      name: $.instanceId
      trace:
        har: true
        domains:
          - "!*global.console.aws.amazon.com*"
          - "!*uxc.*.api.aws*"
          - "!*.prod.pl.panorama.console*"
          - "!*recommendations.widget.console.aws.amazon.com*"
          - "!*telemetry.console.api.aws*"
        video: on-error
      login:
        aws:
          #          connection: connection://aws/production
          login: playwright
          region:
            - eu-west-1
          sessionDuration: 3600
      headless: true
      timeout: 300
      script: |
        const { boot } = require('./playwright-boot');

        let ctx: any;
        async function main() {
          ctx = await boot();
          const { page, log, screenshot, appendChange, writeOutput, close } = ctx;
          const region = process.env.AWS_REGION || 'eu-west-1';

          await ctx.checkLogin(`https://${region}.console.aws.amazon.com/console/home?region=${region}`);

          const rdsUrl = `https://${region}.console.aws.amazon.com/rds/home?region=${region}#databases:`;
          log(`navigating to RDS list: ${rdsUrl}`);
          await page.goto(rdsUrl, { waitUntil: 'domcontentloaded' });
          await page.waitForSelector('a[href*="#database:id="]', { timeout: 30000 });

          const instances: string[] = await page.evaluate(() => {
            const links = document.querySelectorAll('a[href*="#database:id="]');
            return Array.from(links).map(a => {
              const match = (a.getAttribute('href') || '').match(/id=([^&;]+)/);
              return match ? match[1] : '';
            }).filter(Boolean);
          });

          await screenshot('rds_list', { watermark: `RDS instances in ${region}` });
          log(`found ${instances.length} RDS instances: ${instances.join(', ')}`);

          for (const id of instances) {
            const dbUrl = `https://${region}.console.aws.amazon.com/rds/home?region=${region}#database:id=${id};is-cluster=false;tab=configuration`;
            log(`navigating to ${id} configuration tab`);

            // Navigate to about:blank first to force a full page reload (AWS uses hash routing)
            await page.goto('about:blank');
            await page.goto(dbUrl, { waitUntil: 'domcontentloaded' });

            // Wait for this specific instance name to appear in the page heading
            await page.waitForFunction(
              (name: string) => {
                const el = document.querySelectorAll('h1[data-awsui-analytics-label] > span[class*="awsui_heading-text"]')[0];
                return el?.textContent?.includes(name);
              },
              id,
              { timeout: 30000 }
            );
            // Wait for configuration tab content to render
            await page.waitForSelector('[role="tabpanel"] table', { timeout: 15000 }).catch(() => {});

            const configPath = await screenshot(`${id}-configuration`, { watermark: `${id} — Configuration` });
            appendChange({
              change_type: 'RDSConsoleScreenshot',
              id: `rds-config-${id}`,
              config_type: 'AWS::RDS::DBInstance',
              config_id: id,
              summary: `Configuration screenshot for ${id}`,
              screenshot: configPath,
            });

            // Navigate to Maintenance & backups tab
            const backupsUrl = `https://${region}.console.aws.amazon.com/rds/home?region=${region}#database:id=${id};is-cluster=false;tab=maintenance-and-backups`;
            await page.goto('about:blank');
            await page.goto(backupsUrl, { waitUntil: 'domcontentloaded' });
            await page.waitForFunction(
              (name: string) => {
                const el = document.querySelectorAll('h1[data-awsui-analytics-label] > span[class*="awsui_heading-text"]')[0];
                return el?.textContent?.includes(name);
              },
              id,
              { timeout: 30000 }
            );
            await page.waitForSelector('[role="tabpanel"] table', { timeout: 15000 }).catch(() => {});

            const backupsPath = await screenshot(`${id}-backups`, { watermark: `${id} — Maintenance & Backups` });
            appendChange({
              change_type: 'RDSConsoleScreenshot',
              id: `rds-backups-${id}`,
              config_type: 'AWS::RDS::DBInstance',
              config_id: id,
              summary: `Maintenance & backups screenshot for ${id}`,
              screenshot: backupsPath,
            });

          }

          writeOutput(null);
          await close();
        }

        main().catch(async (e) => {
          process.stderr.write(e.stack + '\n');
          if (ctx) {
            await ctx.screenshot('error').catch(() => {});
            await ctx.close({ error: true });
          }
          process.exit(1);
        });

Mapping

Custom scrapers require you to define the id and type for each scraped item. For example, when you scrape a file containing a JSON array, where each array element represents a config item, you must specify the id and type for those items. You can achieve this by using mappings in your custom scraper configuration.

Field	Description	Scheme
`id*`	A static value or JSONPath expression to use as the ID for the resource.	`string` or JSONPath
`name*`	A static value or JSONPath expression to use as the name for the resource.	`string` or JSONPath
`type*`	A static value or JSONPath expression to use as the type for the resource.	`string` or JSONPath
`class`	A static value or JSONPath expression to use as the class for the resource.	`string` or JSONPath
`createFields`	A list of JSONPath expressions used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used.	[]jsonpath
`deleteFields`	A list of JSONPath expressions used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used.	[]jsonpath
`description`	A static value or JSONPath expression to use as the description for the resource.	`string` or JSONPath
`format`	Format of config item, defaults to JSON, available options are JSON, properties. See Formats	`string`
`health`	A static value or JSONPath expression to use as the health of the config item.	`string` or JSONPath
`items`	A JSONPath expression to use to extract individual items from the resource. Items are extracted first and then the ID, Name, Type and transformations are applied for each item.	JSONPath
`status`	A static value or JSONPath expression to use as the status of the config item.	`string` or JSONPath
`timestampFormat`	A Go time format string used to parse timestamps in createFields and deleteFields. (Default: RFC3339)	`string`

Formats

JSON

The scraper stores config items as jsonb fields in PostgreSQL.

Resource providers typically return the JSON used. e.g. kubectl get -o json or aws --output=json.

When you display the config, the UI automatically converts the JSON data to YAML for improved readability.

XML / Properties

The scraper stores non-JSON files as JSON using:

{ 'format': 'xml', 'content': '<root>..</root>' }

You can still access non-JSON content in scripts using config.content.

The UI formats and renders XML appropriately.

Full Scraper Output

When you enable full: true, custom scrapers can return complex objects containing config data, changes, access logs, and external entities.

See the Custom Scraper page for the full output schema, shorthand keys, external entity schemas, and alias resolution.

Scraper​

Playwright​

Trace​

Login​

AWS login​

Browser login​

Mapping​

Formats​

JSON​

XML / Properties​

Full Scraper Output​

Scraper

Playwright

Trace

Login

AWS login

Browser login

Mapping

Formats

JSON

XML / Properties

Full Scraper Output