DevOps Tutorials, Guides & Best Practices

Prometheus Blackbox Exporter is a specialized exporter that performs active probing of HTTP, TCP, DNS, ICMP, gRPC, Unix Socket, and WebSocket endpoints. Unlike regular exporters that expose internal application metrics (see our What is Prometheus and Grafana overview for fundamentals), Blackbox Exporter simulates an external check: it hits the service from the outside and returns Prometheus metrics about availability, response time, TLS certificate status, and DNS records.

🎯 QUICK TAKEAWAY

Use Blackbox Exporter when:

• You need to know if an HTTP endpoint responds with 200 for an external client, not just from inside the pod
• You need SSL/TLS certificate expiry monitoring with alerts at 20/3/0 days before expiry
• DNS records, TCP ports (Redis, PostgreSQL, SMTP), or gRPC services must be checked as a blackbox — without access to internal metrics (DevOps monitoring tools roundup)
• You use Prometheus Operator in Kubernetes — Probe CRD provides native integration without manual scrape_config

Blackbox Exporter Modules and Probers

Blackbox Exporter supports 7 prober types. Each configuration module activates one prober with a set of parameters — timeout, TLS settings, response validation, and so on.

blackbox.yml Configuration

The configuration file blackbox.yml (see the official configuration docs) defines modules — named sets of parameters for different probe types. The top-level structure contains a modules: section where each key is a module name and the value is the prober configuration. The module is selected in Prometheus via the module parameter in the scrape request.

HTTP probe with status code, header, and body validation:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: "ip4"
      valid_status_codes: [200, 201, 204]
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: false
      headers:
        Host: "example.com"
        Accept: "application/json"
      fail_if_body_matches_regexp:
        - "Error"
        - "Service Unavailable"
      fail_if_body_not_matches_regexp:
        - "Welcome"Code language: JavaScript (javascript)

TCP probe with TLS and SSH banner check via query/response:

modules:
  ssh_banner:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - expect: "^SSH-2.0-([^ -]+)(?: (.*))?$"
          labels:
            - name: ssh_version
              value: "${1}"
            - name: ssh_comments
              value: "${2}"

  postgresql:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - send: !!binary AAAACATSFi8=
        - expect_bytes: S
        - starttls: trueCode language: JavaScript (javascript)

DNS probe with response validation via regex:

modules:
  dns_with_validation:
    prober: dns
    timeout: 5s
    dns:
      transport_protocol: "udp"
      preferred_ip_protocol: "ip4"
      query_name: "www.prometheus.io"
      query_type: "A"
      valid_rcodes:
        - NOERROR
      validate_answer_rrs:
        fail_if_matches_regexp:
          - ".*127\.0\.0\.1"
        fail_if_none_matches_regexp:
          - "^(?!127\.).*$"Code language: JavaScript (javascript)

ICMP probe with custom TTL and DF bit:

modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

  icmp_full:
    prober: icmp
    timeout: 10s
    icmp:
      preferred_ip_protocol: "ip4"
      source_ip_address: "10.0.0.1"
      dont_fragment: true
      payload_size: 1500Code language: JavaScript (javascript)

Running via Docker Compose

Blackbox Exporter runs as a sidecar container alongside Prometheus. For ICMP probes on Linux, the NET_RAW capability or network_mode: host is required. On macOS, no extra privileges are needed.

version: "3.8"

services:
  blackbox-exporter:
    image: quay.io/prometheus/blackbox-exporter:v0.28.0
    container_name: blackbox-exporter
    command:
      - '--config.file=/config/blackbox.yml'
      - '--web.listen-address=:9115'
    ports:
      - "9115:9115"
    volumes:
      - ./blackbox.yml:/config/blackbox.yml:ro
    restart: unless-stopped
    cap_add:
      - NET_RAW

  prometheus:
    image: prom/prometheus:v3.6.0
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped
    depends_on:
      - blackbox-exporter

volumes:
  prometheus_data:Code language: PHP (php)

⚠️ Without NET_RAW, ICMP probes are silent

If you see listen ip4:icmp: socket: operation not permitted in the logs, add cap_add: [NET_RAW] or use network_mode: host. In Kubernetes, use securityContext.capabilities.add: [NET_RAW].

Prometheus Scrape Config Setup

Blackbox Exporter uses the multi-target exporter pattern: Prometheus sends an HTTP request to /probe with target and module parameters, and Blackbox Exporter performs the probe on the fly and returns metrics. Three key relabel_configs handle this mechanism:

• __param_target — receives the target URL or IP (from __address__ of static targets)
• instance — the Prometheus label that identifies the target in graphs and alerts (copied from __param_target)
• __address__ — overwritten with the actual Blackbox Exporter address (e.g., blackbox-exporter:9115)

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    scrape_timeout: 15s
    static_configs:
      - targets:
          - https://example.com
          - http://my-app:8080/health
        labels:
          env: 'production'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115Code language: PHP (php)

For different protocols, create separate jobs — blackbox-http, blackbox-tcp, blackbox-dns, blackbox-icmp — with the appropriate module. This lets you filter metrics in PromQL by job and apply different scrape intervals.

# Separate jobs for different protocols
scrape_configs:
  - job_name: 'blackbox-tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - redis-server:6379
          - postgres-server:5432
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - 8.8.8.8
          - 1.1.1.1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115Code language: PHP (php)

Kubernetes: Probe CRD and ServiceMonitor

In Kubernetes with Prometheus Operator, the cleanest way to configure blackbox checks is the Probe CRD. It describes targets, the module, and the Blackbox Exporter address at the individual resource level. Prometheus Operator auto-generates the scrape_config from the Probe.

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: example-com-http
  namespace: monitoring
spec:
  interval: 60s
  scrapeTimeout: 30s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:9115
  targets:
    staticConfig:
      static:
        - http://example.com
        - https://example.com
      labels:
        env: production
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: dns-google
  namespace: monitoring
spec:
  interval: 120s
  module: dns_soa
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:9115
  targets:
    staticConfig:
      static:
        - 8.8.8.8
      labels:
        target_type: dnsCode language: JavaScript (javascript)

To scrape the Blackbox Exporter’s own metrics (health, config reload status), use a standard ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    app: blackbox-exporter
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: http
  selector:
    matchLabels:
      app: blackbox-exporterCode language: PHP (php)

💡 ConfigMap reload

To pick up ConfigMap changes without restarting the pod, add a configmap-reload sidecar container (ghcr.io/jimmidyson/configmap-reload) that sends a POST to /reload on the Blackbox Exporter when the file changes.

Alerting Rules

The key Blackbox Exporter metrics are probe_success (1/0), probe_duration_seconds, probe_http_status_code, and probe_ssl_earliest_cert_expiry. Below are the core alerts for Prometheus:

Example PrometheusRule for basic alerts:

groups:
  - name: blackbox
    rules:
      - alert: BlackboxProbeFailed
        expr: probe_success == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Blackbox probe failed (instance {{ $labels.instance }})"
          description: >
            Probe {{ $labels.module }} for {{ $labels.instance }}
            failed. Check target availability.

      - alert: BlackboxProbeSlow
        expr: probe_duration_seconds > 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Blackbox probe slow (instance {{ $labels.instance }})"
          description: >
            Probe {{ $labels.module }} for {{ $labels.instance }}
            took longer than 1 second ({{ $value }}s).Code language: PHP (php)

Example of three-tier SSL certificate monitoring:

groups:
  - name: blackbox-ssl
    rules:
      - alert: BlackboxSslCertificateExpiringSoon
        expr: 3 <= round((last_over_time(probe_ssl_earliest_cert_expiry[10m]) - time()) / 86400, 0.1) < 20
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate expiring in < 20 days"

      - alert: BlackboxSslCertificateExpiringVerySoon
        expr: 0 <= round((last_over_time(probe_ssl_earliest_cert_expiry[10m]) - time()) / 86400, 0.1) < 3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate expiring in < 3 days"

      - alert: BlackboxSslCertificateExpired
        expr: round((last_over_time(probe_ssl_earliest_cert_expiry[10m]) - time()) / 86400, 0.1) < 0
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate EXPIRED"Code language: JavaScript (javascript)

If the certificate has already expired, Blackbox Exporter by default does not perform the TLS handshake and the probe_ssl_earliest_cert_expiry metric is not exported. To get it even for expired certificates, add tls_config.insecure_skip_verify: true in the HTTP module.

For more details and usage examples, see the official Prometheus Blackbox Exporter guide.

Troubleshooting

Blackbox Exporter vs Alternatives

Blackbox Exporter is the default choice for anyone already running Prometheus. It requires no external services, runs locally, and integrates with Alertmanager without extra layers. Grafana Synthetic Monitoring makes sense when you need geographic distribution and scripted checks. Uptime Kuma is for quick starts without Prometheus. CloudWatch Synthetics — when your entire infrastructure is in AWS and you need complex browser canaries.

FAQ