DevOps Tutorials, Guides & Best Practices

Prometheus and Grafana solve different parts of monitoring. Prometheus collects, stores, and queries metrics. Grafana queries data sources and turns their data into dashboards, exploration views, and alerts. Most teams do not choose one instead of the other. They use Prometheus as the metrics backend and Grafana as the interface people use to investigate and communicate what the metrics show.

Short recommendation: choose Prometheus when you need a metrics pipeline and PromQL-based rules. Choose Grafana when the data already exists and you need dashboards or cross-source exploration. Use both when Prometheus owns metrics and Grafana is the shared operational view.

For a broader introduction before the architecture decision, read What Is Prometheus and Grafana?.

Prometheus vs Grafana at a glance

The useful comparison is ownership, not feature count.

This is why “Prometheus or Grafana?” is usually the wrong architecture question. Ask two questions instead:

1. Which system owns the metric samples and query API?

2. Which system owns the human workflow for dashboards, investigations, and alerts?

How the data flows

A common open-source path looks like this:

Application / exporter
        |
        | exposes /metrics
        v
Prometheus scraper -> Prometheus TSDB -> PromQL
                                      |
                                      v
                                Grafana data source
                                      |
                         dashboards / Explore / alertsCode language: plaintext (plaintext)

Prometheus identifies each time series by a metric name and a set of labels. It pulls samples from configured targets, stores them, and evaluates PromQL queries. Grafana sends queries to the Prometheus HTTP API and renders the returned series. Grafana can also query Loki, Elasticsearch, SQL databases, cloud monitoring services, tracing backends, and many other sources.

The boundary matters during incidents. If a Grafana panel is empty, the failure could be in instrumentation, target discovery, scraping, storage, the PromQL query, Grafana data-source configuration, dashboard variables, or the panel itself. Treating Grafana as if it collected the metric hides half of that diagnostic chain.

What Prometheus owns

Prometheus is the metrics system in this pair. Its core responsibilities are:

– discovering or receiving the list of scrape targets;

– pulling metrics over HTTP at configured intervals;

– storing timestamped samples in a labeled time-series model;

– running PromQL queries;

– precomputing expensive or reused expressions with recording rules;

– evaluating alerting rules;

– sending firing alerts to Alertmanager or another compatible receiver.

For external endpoint checks, the Prometheus Blackbox Exporter guide covers the exporter-specific probe path. This comparison stays focused on the ownership boundary between Prometheus and Grafana.

A minimal scrape configuration makes that ownership visible:

# prometheus.yml
scrape_configs:
  - job_name: api
    scrape_interval: 15s
    static_configs:
      - targets:
          - api:8080Code language: YAML (yaml)

After the target is healthy, PromQL turns raw counters and histograms into operational signals:

# Per-second request rate over the last five minutes
sum by (service) (
  rate(http_requests_total[5m])
)Code language: plaintext (plaintext)

# 95th-percentile request duration by service
histogram_quantile(
  0.95,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)Code language: plaintext (plaintext)

Prometheus includes local storage, but local storage is not a universal long-term or multi-region architecture. Define retention, disk capacity, failure tolerance, and recovery expectations. Add remote write, remote storage, federation, or a compatible distributed backend only when those requirements are explicit.

For a concrete example of Prometheus metrics driving diagnosis, the Kubernetes CPU throttling guide uses rate and ratio signals to separate CPU limits from general application slowness.

What Grafana owns

Grafana is the human-facing query and visualization layer in this pair. A Grafana data source contains the connection and query configuration for a backend. Panels use those data sources to retrieve results and transform them into graphs, tables, gauges, heatmaps, logs, traces, or other views.

Grafana is most valuable when teams need to:

– share one dashboard instead of copying PromQL links;

– explore a metric interactively without editing the production rule set;

– use template variables for service, cluster, region, or environment;

– correlate metrics with logs, traces, annotations, and deployment events;

– manage access to dashboards and folders;

– query more than one backend from the same operational interface.

The Prometheus data source is built into Grafana. It supports PromQL and Prometheus-compatible query APIs such as Grafana Mimir and Thanos. Configuring it does not move the samples into Grafana. It gives Grafana a query path to the backend.

A useful dashboard is not a wall of every available metric. Start from an operational decision: Is the service available? Is latency outside its objective? Is demand saturating a constrained resource? Did a deployment change the signal? The broader DevOps monitoring tools guide explains how metrics fit beside logs, traces, and user-facing checks.

Prometheus alerting vs Grafana Alerting

Both products can evaluate alert conditions, but they create two different ownership models.

Prometheus rules and Alertmanager

Prometheus evaluates PromQL alerting rules close to the metric backend. A rule can use the same recording rules and labels as dashboards and ad hoc queries. Prometheus sends firing alerts to Alertmanager, which handles grouping, routing, silencing, inhibition, and notification integrations.

groups:
  - name: api-slo
    rules:
      - alert: ApiHighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
          > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: API error rate is above 5 percentCode language: YAML (yaml)

This model fits teams that keep metric rules in version control and want Prometheus to own evaluation even if Grafana is unavailable.

Grafana-managed alerting

Grafana Alerting can evaluate queries and expressions from several supported data sources and manage rules, contact points, and notification policies in a consolidated workflow. It is useful when operations teams need one alerting interface across Prometheus metrics and other backends.

Do not implement the same incident condition independently in Prometheus and Grafana without a deliberate deduplication design. Duplicate rules drift, page twice, and make it unclear which rule is authoritative.

An alert still needs a diagnostic path. Metrics can show memory pressure, while events and logs explain why a container exited. The Kubernetes OOMKilled guide demonstrates that handoff from signal to cause.

When Prometheus alone is enough

Use Prometheus without Grafana when:

– a service needs metric scraping, storage, PromQL, and alert rules;

– dashboards are not a current requirement;

– a small engineering team can work with PromQL and the expression browser;

– another interface already queries the Prometheus API;

– minimizing components is more valuable than rich visualization.

This is a valid small-system architecture. Do not add Grafana only because it appears in a reference stack diagram.

When Grafana without Prometheus is enough

Use Grafana without Prometheus when your data already lives elsewhere, such as:

– a managed cloud monitoring service;

– Grafana Mimir or another Prometheus-compatible backend;

– Loki for logs or Tempo for traces;

– Elasticsearch;

– PostgreSQL, MySQL, or another supported database;

– an existing metrics platform with a Grafana data-source plugin.

Grafana does not require Prometheus. It requires a data source that can answer the queries used by panels, Explore, and alerts.

This distinction also prevents a common observability mistake: logs are not Prometheus metrics. If the investigation requires container stdout or daemon logs, use the relevant log backend and a workflow such as the Docker logs command guide, then correlate that evidence with metrics in Grafana.

When to use both

Use Prometheus and Grafana together when you want an open metrics pipeline plus a shared operational interface. This is the default recommendation for many Kubernetes and service-oriented environments because the responsibilities stay clear:

1. Applications and exporters expose metrics.

2. Prometheus discovers targets and scrapes samples.

3. Prometheus stores the time series and evaluates PromQL.

4. Grafana queries Prometheus for dashboards and exploration.

5. One explicitly chosen alerting path owns each incident rule.

For example, a Kubernetes dashboard can show CPU usage, throttling ratio, memory working set, restart counts, and request latency. A drill-down should then link to the relevant runbook rather than pretending the graph diagnoses the root cause by itself.

A practical setup sequence

Build the smallest path that proves collection, query correctness, visualization, and alert ownership before adding high-availability or long-retention components.

1. Define the decision before the dashboard

Write the operational question and owner first. “Page the API owner when sustained server errors exceed the service threshold” is implementable. “Monitor the API” is not.

2. Instrument the service and verify the raw target

Confirm the /metrics endpoint, labels, metric types, and target health in Prometheus. Avoid unbounded labels such as user IDs, request IDs, or raw URLs because cardinality grows the number of stored series.

3. Verify the PromQL outside the dashboard

Run the query against Prometheus, check the label set, and test empty or missing-series behavior. A dashboard should not be the first place a query is validated.

4. Configure the Prometheus data source in Grafana

Use the Prometheus URL reachable from the Grafana server, not necessarily the URL reachable from your laptop. Verify data-source health and access mode before building panels.

5. Build one decision-oriented dashboard

Start with service availability, traffic, errors, latency, and saturation. Add deployment annotations or links to release records. Use variables only where they reduce duplication without hiding which environment is being viewed.

6. Choose one owner for each alert

Decide whether Prometheus rules or Grafana-managed rules own the condition. Record the rule, routing path, runbook, severity, and test procedure.

7. Test failure and recovery

Generate a bounded failure, confirm the metric changes, verify the alert transitions through pending and firing states, check notification routing, and prove that the dashboard and runbook help the responder. A green configuration file is not an alerting test.

Storage and scaling decisions

Prometheus and Grafana scale along different dimensions.

Do not call Grafana a scaling layer for Prometheus storage. A faster dashboard cannot repair excessive metric cardinality, insufficient TSDB capacity, or an expensive PromQL query. Likewise, adding Prometheus replicas does not organize dashboard ownership or access control.

Decision examples

– Small internal API: Prometheus alone may be enough for scraping, a few PromQL checks, and alerts.

– Kubernetes platform team: Prometheus plus Grafana is usually the practical starting point for shared metrics, dashboards, and investigation.

– Company already on CloudWatch or Azure Monitor: Grafana can provide a shared interface without deploying Prometheus solely for visualization.

– Multi-source operations center: Grafana can centralize views and selected alerts across metrics, logs, traces, and cloud services, while each backend retains its storage role.

– Compliance-heavy rule ownership: Prometheus rules in version control plus Alertmanager may provide the clearer review trail for metric alerts.

Common mistakes

– Treating the tools as substitutes: one owns metrics; the other primarily queries and presents data.

– Assuming Grafana stores Prometheus samples: the configured backend still owns ingestion, retention, and availability.

– Duplicating alerts in both systems: two rule engines create drift and duplicate notifications.

– Building dashboards before validating metrics: polished panels cannot fix missing targets, wrong labels, or invalid PromQL.

– Using high-cardinality labels: identifiers such as user ID or request ID can make the Prometheus series count explode.

– Mixing metrics and logs without ownership: a graph and a log search can appear in one Grafana workflow, but they still come from different backends.

– Leaving alert routing untested: a firing expression is not enough; grouping, contact points, silence behavior, and runbook links need tests.

– Using one dashboard for every audience: executives, service owners, and incident responders need different decisions and time ranges.

Sources

Prometheus overview

Prometheus data model

Prometheus storage

Prometheus alerting overview

Grafana data sources

Grafana Prometheus data source

Grafana dashboards

Grafana Alerting

FAQ

These answers cover the ownership questions that most often cause duplicate components or unclear alerting paths.