Prometheus Interview Questions 2026: Prometheus vs Grafana vs Datadog Compared

Prometheus interview questions come up in almost every DevOps and SRE loop, usually followed by "why not Grafana or Datadog?" Candidates need to explain how Prometheus scrapes and stores metrics, write correct PromQL, and argue the trade-offs against a managed platform. Interviewers use this to tell real operational experience apart from textbook knowledge. The Prometheus and monitoring interview module has the same topics as drill questions.

Prometheus Interview Questions Asked in DevOps and SRE Rounds

These are the Prometheus-specific questions that show up most often, with the answer a confident candidate gives. They assume Prometheus 3.x: the current stable line is 3.15 (September 2026), with 3.16 in release candidate.

Q: Why does Prometheus pull metrics instead of receiving pushes?

Pulling makes target health observable: every scrape produces an up series, so a dead target shows up as up == 0 instead of silence. The server controls the scrape rate, which protects it from a misbehaving client, and a target can be checked by hand by opening its /metrics endpoint in a browser. The cost is that Prometheus must reach every target over the network, which is why short-lived batch jobs need the Pushgateway and why cross-network setups use remote write.

Q: What are the four metric types, and when is a summary the wrong choice?

Counter (only goes up, resets on restart), gauge (goes up and down), histogram (observations counted into buckets) and summary (quantiles computed in the client). A summary's quantiles cannot be aggregated: averaging the p99 of ten pods does not give the p99 of the service. Histograms can be summed across instances first and turned into a quantile on the server, which is why they are the default for latency.

# Classic histogram: aggregate the buckets, keep the le label
histogram_quantile(0.99,
  sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))

# Native histogram: no _bucket suffix, no le label
histogram_quantile(0.99,
  sum by (job) (rate(http_request_duration_seconds[5m])))

Forgetting le in the by clause of the classic form is the most common PromQL mistake in live coding rounds: the query returns nothing usable.

Q: What is the difference between rate(), irate() and increase()?

rate() returns the per-second average over the whole range and handles counter resets. irate() uses only the last two samples in the range, so it reacts fast but is noisy, which makes it fine for a zoomed-in graph and wrong for alerting. increase() is rate() multiplied by the range duration. Because Prometheus extrapolates to the window boundaries, increase() on an integer counter can return a non-integer, and interviewers like to ask why.

Q: What is cardinality, and how is a cardinality explosion diagnosed?

Every unique combination of metric name and label values is a separate time series held in memory. A label like user_id, email or a raw URL path containing IDs multiplies the series count until the server runs out of memory. Diagnosis starts on the TSDB status page (/tsdb-status) or with a query:

# Top 10 metric names by number of series
topk(10, count by (__name__) ({__name__=~".+"}))

The fix is at the source: drop or rewrite the label with metric_relabel_configs, or set sample_limit on the scrape job so one bad deployment cannot take the server down.

Q: What are recording rules for?

They precompute an expensive expression at each evaluation interval and store the result as a new series. Dashboards and alerts then query the cheap precomputed series. The naming convention is level:metric:operations:

groups:
  - name: api-recording-rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Q: What does the for clause in an alerting rule do?

An alert whose expression becomes true moves from inactive to pending, and only reaches firing once the condition has held for the for duration. This filters one-scrape blips. keep_firing_for does the reverse: it keeps the alert firing for a while after the condition clears, which stops flapping alerts from paging twice.

Q: How is Prometheus made highly available?

Run two identical replicas scraping the same targets, each with a distinct replica external label. Both send alerts to an Alertmanager cluster, which deduplicates notifications through its gossip protocol. For queries, Thanos Query or Mimir deduplicates the two series by ignoring the replica label. Prometheus itself has no clustering: a single server's local TSDB is not replicated.

Q: Federation or remote write for multi-cluster setups?

Federation lets a global Prometheus scrape selected, usually aggregated, series from the /federate endpoint of each cluster's Prometheus. It suits a small set of rollups. Remote write streams every sample to a central store such as Mimir, Thanos Receive or a vendor backend, and is the standard answer for global querying and long-term retention.

Q: How is a target that disappeared from service discovery detected?

up == 0 only fires while the target is still discovered. If the pod or the service discovery entry is gone, the series stops existing and the alert never fires. absent(up{job="api-server"}) covers that case.

Q: What changed in Prometheus 3?

Prometheus 3.0 (November 2024) shipped a new UI, allowed UTF-8 characters in metric and label names, and added a built-in OTLP receiver at /api/v1/otlp/v1/metrics, enabled with --web.enable-otlp-receiver. Native histograms became stable in 3.8. Since 3.9, the old feature flag does nothing and ingestion requires scrape_native_histograms: true in the scrape or global config. 3.15 made XOR2 float chunk encoding stable and added OpenMetrics 2.0 scraping.

Architecture and Data Model: How Each Tool Handles Metrics

Once the Prometheus fundamentals are covered, interviewers usually move to how the three tools differ architecturally.

Prometheus uses a pull-based model. It scrapes HTTP endpoints (typically /metrics) at configured intervals, storing time-series data in a custom local TSDB. Each time series is identified by a metric name and a set of key-value labels. The 3.x line adds stable native histograms, an OTLP receiver and Remote Write 2.0 for shipping samples to central storage.

Grafana does not collect or store metrics. It connects to data sources, Prometheus, Loki, Tempo, InfluxDB, Elasticsearch and 100+ others, then renders dashboards from their data. Grafana Labs also maintains Mimir (long-term metrics storage), Loki (log aggregation) and Tempo (distributed tracing), which together with Grafana form the open-source LGTM stack. Grafana 13, announced at GrafanaCON in April 2026, made Git Sync for dashboards generally available, turned dynamic dashboards on by default and removed the grafana-cli and grafana-server commands.

Datadog operates as a push-based SaaS. Agents installed on hosts push metrics, logs and traces to Datadog's cloud backend. Everything, ingestion, storage, querying, alerting and dashboards, lives inside a single managed platform. The Watchdog engine flags anomalies automatically, without manual threshold configuration.

# prometheus.yml - Pull-based scrape configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api-server'
    scrape_native_histograms: true
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

This YAML defines Prometheus's pull model: it discovers Kubernetes pods via service discovery, keeps only those annotated with prometheus.io/scrape: "true", rewrites the address to the annotated port and scrapes /metrics every 15 seconds. The second rule joins both source labels with ;, which is why the regex matches host:port;port.

PromQL vs Datadog Query Language: Syntax and Capabilities

Query languages are a frequent interview topic. Candidates should demonstrate fluency in at least one, and explain the trade-offs between them.

PromQL (Prometheus Query Language) is the standard for metrics queries across the Prometheus and Grafana ecosystem. It supports instant vectors, range vectors, aggregation operators and recording rules.

# Request rate per service over 5 minutes
rate(http_requests_total{job="api-server"}[5m])

# 99th percentile latency using native histograms
histogram_quantile(0.99, rate(http_request_duration_seconds[5m]))

# Error rate as percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# Predict disk full in 4 hours using linear regression
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

Datadog's query language uses a different syntax built around functions and scoping:

# Equivalent request rate in Datadog
sum:http.requests{service:api-server}.as_rate()

# Anomaly detection (no built-in PromQL equivalent)
anomalies(avg:system.cpu.user{service:api-server}, 'agile', 3)

# Forecast query
forecast(avg:system.disk.free{host:web-01}, 'linear', 1)

PromQL has deeper expressiveness for ad-hoc analysis. Datadog's query language trades some flexibility for built-in algorithmic functions like anomalies() (algorithms basic, agile, robust) and forecast() (linear, seasonal) that would require external tooling in a Prometheus stack.

Alerting Strategies: Rules-Based vs ML-Powered Detection

Alerting philosophy differs sharply between these tools, and understanding these differences demonstrates operational maturity in interviews.

Prometheus Alertmanager receives alerts from rules that Prometheus evaluates at fixed intervals, then routes them through a configurable pipeline with grouping, silencing and inhibition:

# alert-rules.yml - Prometheus alerting rules
groups:
  - name: api-server-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5..",job="api-server"}[5m]))
          / sum(rate(http_requests_total{job="api-server"}[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 5% for 5 minutes"
          runbook: "https://wiki.internal/runbooks/high-error-rate"

      - alert: PodMemoryPressure
        expr: |
          container_memory_working_set_bytes{namespace="production"}
          / container_spec_memory_limit_bytes{namespace="production"} > 0.9
        for: 10m
        labels:
          severity: warning

This approach requires operators to define explicit thresholds. The advantage is full transparency: every alert has a reviewable expression. The drawback is threshold fatigue, since static values often need seasonal adjustments.

Grafana Alerting (the unified alerting system, default since Grafana 9) evaluates queries across any connected data source and supports multi-dimensional alerts with notification policies.

Datadog Monitors combine static thresholds with anomaly, outlier and forecast monitors. Watchdog automatically flags performance anomalies without manual rule creation. This reduces configuration overhead at the cost of less transparency into detection logic.

Pricing and Total Cost of Ownership in 2026

Pricing is a deciding factor in real-world tool selection and comes up frequently in system design interviews. Understanding cost structures demonstrates business awareness. Datadog figures below are list prices on annual billing.

The open-source stack appears cheaper on paper, but requires dedicated engineering time for upgrades, capacity planning and high-availability configuration. Datadog's managed model shifts that burden to the vendor. For teams with fewer than 5 engineers running infrastructure, managed solutions often have lower total cost when engineering time is factored in.

Kubernetes Monitoring: Integration Depth Compared

Prometheus is the default metrics system of the Kubernetes ecosystem, which makes this the dominant interview topic for container orchestration monitoring.

Prometheus + Grafana integrates natively with Kubernetes through the kube-prometheus-stack Helm chart, which deploys Prometheus Operator, Alertmanager, node-exporter, kube-state-metrics and pre-built Grafana dashboards in a single installation:

# Deploy full monitoring stack on Kubernetes
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

This deploys a production-grade monitoring stack with persistent storage and 30-day retention. The Prometheus Operator uses Custom Resource Definitions (ServiceMonitor, PodMonitor) to declaratively configure scrape targets. Overriding chart values this way is covered in more depth in the Helm charts interview guide.

Datadog deploys an agent DaemonSet and a Cluster Agent. The Cluster Agent handles API server communication centrally, reducing load on the Kubernetes API. Datadog's Live Containers view provides real-time visibility into pod states, and Orchestrator Explorer maps relationships between deployments, services and pods.

For teams already invested in the Kubernetes ecosystem, the Prometheus-native approach avoids introducing an external dependency. Teams prioritizing speed of setup with less operational investment lean toward Datadog.

OpenTelemetry and Vendor Neutrality

OpenTelemetry (OTel) has become the industry standard for instrumentation, and interviewers increasingly ask about it alongside monitoring tool comparisons.

Prometheus 3.x accepts OTLP metrics directly once --web.enable-otlp-receiver is set, so a Collector is no longer required as an intermediary. Grafana Alloy (the successor to Grafana Agent) serves as both an OTel Collector and a Prometheus scraper. Datadog supports OTLP ingestion but recommends its proprietary agents for "full feature access," creating a soft lock-in.

OTel adoption matters because it decouples instrumentation from backend choice. Application code instrumented with OTel SDKs can send telemetry to Prometheus, Grafana Cloud, Datadog or any compatible backend without code changes. This flexibility is a strong argument in system design interviews when discussing long-term observability strategy.

Observability Interview Questions Comparing the Three Tools

Beyond Prometheus itself, these questions test whether a candidate can reason across tools. Each answer highlights the concept the interviewer is checking.

Q: Explain the difference between monitoring, observability, and alerting.

Monitoring tracks predefined metrics and checks known failure modes. Observability enables investigation of unknown failure modes through metrics, logs and traces (the "three pillars"). Alerting triggers notifications when conditions breach defined thresholds or anomaly baselines. Monitoring answers "is the system healthy?" Observability answers "why is the system unhealthy?"

Q: When would Prometheus be a poor choice for monitoring?

Prometheus is optimized for reliability over durability: it prioritizes availability of the monitoring system itself. Scenarios where Prometheus struggles: long-term storage beyond 30 days (requires Thanos, Mimir or Cortex), per-request billing data requiring 100% accuracy (scrape-based collection can miss samples), and event-based systems that need push-based collection (the Pushgateway exists, but only for service-level batch jobs).

Q: How does Grafana's "Big Tent" philosophy affect observability architecture?

Grafana connects to any data source without requiring data migration. This means teams can query Prometheus, Elasticsearch, CloudWatch and Datadog from a single dashboard. The trade-off is operational complexity: maintaining multiple backends requires more infrastructure expertise than a single-vendor approach. Interviewers test whether candidates can articulate this trade-off clearly.

Q: What is Datadog's high-watermark billing, and why does it matter?

Datadog meters host count hourly, drops the top 1% of hours, and bills at the 99th-percentile peak. This means temporary auto-scaling spikes (Black Friday traffic, for instance) inflate the monthly bill even after instances are terminated. Candidates who mention this demonstrate real operational experience with cost management, which SRE roles increasingly require.

Q: How would an SLO-based alerting strategy differ between Prometheus and Datadog?

In Prometheus, SLO alerting uses recording rules to pre-compute error budgets and burn rate alerts (the multi-window, multi-burn-rate approach from Google's SRE workbook). Datadog offers built-in SLO widgets and monitors that track burn rate automatically. Both approaches implement the same concept, but Prometheus requires more manual configuration while Datadog provides a managed workflow. Candidates should reference burn rate windows (1h, 6h, 3d) and error budget consumption rates in their answer.

Decision Framework: Choosing the Right Stack

The right choice depends on three variables: team size, operational maturity and budget constraints. There is no universally correct answer, and interviewers expect candidates to reason through trade-offs rather than declare a winner.

Sources

• Prometheus releases: 3.15.0 (September 2026) stabilizes XOR2 chunk encoding, adds OpenMetrics 2.0 scraping and deprecates --log.level
• Prometheus native histograms specification: stable since 3.8, scrape_native_histograms required since 3.9
• What's new in Grafana 13.0: Git Sync GA, dynamic dashboards by default, breaking changes
• Datadog list pricing: infrastructure, APM and log prices used in the cost table
• Datadog algorithmic functions: anomalies() and forecast() syntax

What to Remember for a Prometheus or Observability Interview

• Prometheus is the standard for metrics collection in Kubernetes environments; know pull vs push, the four metric types, rate() vs irate(), cardinality and HA by heart
• On Prometheus 3.9 and later, native histograms are only ingested when scrape_native_histograms: true is set
• Grafana is a visualization layer, not a metrics database: it connects to 100+ data sources including Prometheus, and the LGTM stack (Loki, Grafana, Tempo, Mimir) forms a complete open-source observability platform
• Datadog provides the fastest path to full-stack observability with built-in anomaly detection, at the cost of higher pricing and vendor lock-in
• OpenTelemetry adoption makes backend choice less permanent: instrumentation stays the same regardless of which tool stores and queries the data
• In interviews, demonstrate ability to reason about trade-offs (cost, control, complexity) rather than advocate for a single tool
• For broader preparation, the essential DevOps interview questions guide covers the adjacent topics interviewers chain onto monitoring