Skip to main content

Observability

Introduction​

This configuration is ideal for development teams requiring a complete testing environment with comprehensive monitoring. It deploys the default CDK OP Reth setup enhanced with debugging and observability tools, making it perfect for development, testing, and troubleshooting.

What's Deployed?​

Use Cases​

  • Teams needing comprehensive monitoring and debugging tools
  • Testing and troubleshooting rollup environments
  • Development environments requiring full visibility into blockchain activity

Deployment​

params.yml
args:
  additional_services:
    - blockscout
    - status_checker
    - observability

To deploy this environment:

kurtosis run --enclave cdk --args-file params.yml .

After deployment, retrieve service URLs with:

kurtosis port print cdk prometheus-001 http
kurtosis port print cdk grafana-001 http

Open the printed URLs in your browser to access metrics and dashboards.

Metrics​

Adding a service that emits Prometheus metrics to kurtosis-cdk is straightforward: ensure the service has a prometheus port configured.

plan.add_service(
    name="service-a",
    config=ServiceConfig(
        ports={"prometheus": PortSpec(9090, application_protocol="http")},
    ),
)

To verify Prometheus is ingesting metrics correctly:

  1. Run:

    curl $(kurtosis port print cdk panoptichain-001 prometheus)/metrics

    Adjust the metrics path if necessary for the service's Prometheus configuration.

  2. Navigate to:

    $(kurtosis port print cdk prometheus-001 http)/targets

    Confirm the service appears and query a service-specific metric.

Dashboards​

Several predefined dashboards are available, some with filters for network, provider, or job.

dashboards

The Panoptichain dashboard displays metrics from Panoptichain, an open-source blockchain monitoring tool that captures data via RPC calls. All Panoptichain metrics are prefixed with panoptichain_. View all available Panoptichain metrics here.

panoptichain

The Services dashboard shows metrics for all services in kurtosis-cdk. When adding a new service with Prometheus metrics, add a row here or create a dedicated dashboard.

services

Saving Dashboards​

By default, Grafana dashboard changes do not persist across kurtosis runs. To save a dashboard:

  1. Share → Export → Save to file. Ensure Export for sharing externally is unchecked.
  2. Save the file to: static_files/additional_services/grafana/dashboards
  3. Restart Grafana or kurtosis-cdk.

Status Checks​

Status checks are scripts that assess network health. They reside in static_files/additional_services/status-checker/checks.

Writing Checks​

Place each status-check script in the checks directory with a proper shebang (e.g. #!/usr/bin/env bash) on the first line so it can be executed directly. A script passes when it exits with code 0 and is considered failed on any non-zero exit code. Reference existing checks to maintain consistency and minimize false positives. Because status-check scripts are stateless and ephemeral between executions, required state must be managed externally. File-based storage or environment variables are suitable for simple state management, while a database should be used for more complex scenarios.

block-number.sh
#!/usr/bin/env bash
cast block-number --rpc-url $L1_RPC_URL

The container running status checks provides environment variables:

env_vars = {
    "L1_RPC_URL": args.get("l1_rpc_url"),
    "L2_RPC_URL": l2_rpc_url,
    "SEQUENCER_RPC_URL": sequencer_rpc_url,
    "CONSENSUS_CONTRACT_TYPE": args.get("consensus_contract_type"),
}

Viewing Results​

To view status check results, run:

kurtosis service logs cdk status-checker-001 --follow

Logs show each check's result:

[status-checker-001] {"level":"info","check":"l2-coinbase.sh","success":true,"time":"2025-06-17T19:24:20Z"}

The status checker also appears in the Grafana Services dashboard:

status-checker

Alerting​

Alerting monitors Prometheus-ingested metrics and triggering notifications when defined thresholds are breached. A set of preconfigured alerting rules is included out of the box. It is recommended to use alerting for simple use cases where there are existing Prometheus metrics and the status-checker for more complex scenarios.

alerting

Slack Notifications​

By default, Slack notifications are disabled. To enable them, set the following environment variables in grafana.star:

  • SLACK_WEBHOOK_URL
  • SLACK_CHANNEL
  • SLACK_USERNAME

When enabled, Grafana will post alert messages to the specified Slack channel during a Kurtosis run.

Exporting Rules​

To export and persist alerting rules:

  1. In Grafana, go to Alerting → Alert Rules.
  2. Click Export → YAML → Download.
  3. Save the downloaded file as static_files/additional_services/grafana/alerting.yml.tmpl.

alerting-export