Tools / Grafana Interview questions

Grafana is an open-source observability and data visualization platform. You point it at one or more data sources, write queries, and turn the results into dashboards, explorations and alerts.

Grafana does not store your metrics itself. It queries the systems that do, such as Prometheus, Loki, InfluxDB or a SQL database, and renders what comes back. The backend is written in Go and the UI in TypeScript and React.

Teams use it mostly for infrastructure and application monitoring, but the same dashboards also work for business KPIs, IoT data and CI/CD metrics.

Grafana bundles several tools around one idea: query data from anywhere and make it useful. The ones interviewers usually expect you to name:

• Dashboards built from panels on a shared time range
• Explore for ad hoc queries without saving a dashboard
• Alerting with rules, contact points and notification policies
• Template variables for reusable dashboards
• Annotations to mark deployments and incidents on graphs
• Transformations to reshape query results
• Plugins for panels, data sources and apps
• Provisioning and RBAC for running it at scale

Grafana ships with built-in support for dozens of data sources, and plugins extend that list further. Each one has its own query editor.

You can also combine several of them in a single panel using the Mixed data source.

A dashboard is a set of panels arranged on a grid, sharing a time range, refresh interval and optional variables. It is the main way people look at data in Grafana.

Under the hood a dashboard is just a JSON document stored in the Grafana database. It describes each panel, its queries, layout, variables and settings, which is why dashboards are easy to export, version in Git and provision.

Dashboards live inside folders, and rows can group and collapse related panels.

A panel is the basic visual building block of a dashboard. Each panel holds one or more queries, optional transformations, a visualization type, and display settings.

Display settings, called field options, cover units, decimals, thresholds, value mappings, colors and overrides for specific series. A panel can also carry data links and alert-related context.

You build panels in the panel editor, which shows a live preview as you tweak queries and options.

Choosing the right panel matters more than people think. These are the types that come up most often:

Grafana listens on port 3000. A fresh install has the username admin and password admin, and the UI asks you to change the password on first login.

For automated installs, set the admin password with an environment variable instead. This Docker example starts Grafana with a custom admin password:

docker run -d -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=S3cretPass \
  grafana/grafana

Grafana is configured through an INI-style file called grafana.ini. On Linux package installs it lives at /etc/grafana/grafana.ini. Defaults come from defaults.ini, so you only override what you need.

Settings are grouped in sections such as [server], [database], [security] and [auth]. Any setting can also be overridden with an environment variable named GF_<SECTION>_<KEY>, and environment variables win over the file.

[server]
http_port = 3000

# same thing as an environment variable:
# GF_SERVER_HTTP_PORT=3000

Restart Grafana after changing the file.

Template variables are dropdowns at the top of a dashboard that get substituted into queries. Instead of hard-coding a server name, you write $instance and let the viewer pick.

The common types are Query (options come from a data source), Custom (a fixed list), Interval, Data source, Text box, Constant and Ad hoc filters. A variable can allow multiple selections or an All option.

# Query variable definition for Prometheus
label_values(up, instance)

# used in a panel query
rate(node_cpu_seconds_total{instance=~"$instance"}[5m])

Explore is a query workspace for ad hoc investigation. You pick a data source, run queries, and see results without creating or saving a dashboard.

It is built for troubleshooting. You can open a split view to compare two queries or two data sources side by side, for example metrics on the left and logs for the same time range on the right. Query history lets you re-run earlier queries, and any query can be added to a dashboard once it looks right.

Annotations are markers drawn on time series graphs that tie an event to a point in time, such as a deployment, a config change or an incident.

You can add them manually by Ctrl/Cmd-clicking a graph, pull them from a data source with an annotation query, or create them through the HTTP API from a CI pipeline. The built-in Annotations & Alerts query also shows alert state changes.

When a metric jumps, seeing a deployment marker at the same moment often answers the question immediately.

Grafana supports three plugin types. Each extends a different part of the product.

Plugins are signed by Grafana Labs or the community. Unsigned plugins are blocked unless you list them in allow_loading_unsigned_plugins.

Grafana comes in three flavors. They share the same core, so dashboards move between them easily.

Loki is Grafana's log aggregation system, often described as Prometheus for logs. Instead of indexing the full text of every line, it indexes only a small set of labels per log stream and stores compressed chunks in cheap object storage.

That design keeps storage and indexing costs low. You query with LogQL, which selects streams by label and then filters or parses the lines.

{app="checkout", env="prod"} |= "error" | json | level="error"

Logs reach Loki through an agent such as Grafana Alloy.

Tempo is a distributed tracing backend. It stores traces in object storage and, unlike older tracing systems, does not need a heavy index. You mostly find traces by trace ID, usually jumping from a log line or a metric exemplar.

It ingests common protocols, including OpenTelemetry (OTLP), Jaeger and Zipkin. For searching by attributes there is TraceQL, for example finding slow spans in one service.

{ resource.service.name = "checkout" && duration > 500ms }

LGTM is Grafana Labs' open-source observability stack. Each letter is one component and one signal type.

Together they let you go from a metric spike to the related logs and traces inside one UI.

Alloy is Grafana's distribution of the OpenTelemetry Collector. It collects metrics, logs, traces and profiles and forwards them to Prometheus, Mimir, Loki, Tempo or any OTLP backend. It replaces Grafana Agent and Promtail.

It is configured with components wired together in Alloy's own syntax. This one scrapes an app and writes the metrics to Mimir:

prometheus.scrape "app" {
  targets    = [{"__address__" = "localhost:8080"}]
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint { url = "http://mimir:9009/api/v1/push" }
}

There are several options depending on who needs access and how live the data should be.

• Share link: a URL that keeps the time range and variable values; viewers still need access
• Snapshot: a static copy with the data embedded, so no data source access is required
• Public dashboard: read-only access without logging in
• Export JSON: move the dashboard to another instance or Git
• Embed: iframe, which needs allow_embedding = true
• Reports: scheduled PDFs in Enterprise and Cloud

Access in Grafana is organization-based, with three basic roles per org plus an instance-wide admin flag.

Folder and dashboard permissions, and teams, refine these roles further.

The JSON model is the full definition of a dashboard. You can see it under Dashboard settings, copy it, and import it elsewhere.

Key fields are uid (a stable identifier used in URLs and provisioning), title, panels, templating, time, annotations, and schemaVersion. The numeric id is internal and should be set to null when importing.

{
  "uid": "node-overview",
  "title": "Node overview",
  "time": { "from": "now-6h", "to": "now" },
  "templating": { "list": [] },
  "panels": []
}

From the UI, open Connections > Data sources, click Add new data source, and choose a type. Then enter the connection details and press Save & test.

1. Pick the data source type, for example Prometheus
2. Enter the URL, such as http://prometheus:9090
3. Set authentication or TLS options if needed
4. Click Save & test and confirm the green success message

Admin rights are required. For repeatable setups, define data sources in a provisioning YAML file instead.

The simplest way is the Grafana CLI, followed by a restart. Plugins land in /var/lib/grafana/plugins on Linux.

grafana cli plugins install grafana-clock-panel
sudo systemctl restart grafana-server

In containers, pass plugin IDs through GF_INSTALL_PLUGINS. You can also browse and install from Administration > Plugins in the UI. Unsigned plugins need to be allowed explicitly in the config.

Go to Dashboards > New > Import. You can paste JSON, upload a JSON file, or enter the ID of a dashboard from grafana.com. For example, ID 1860 is the popular Node Exporter Full dashboard.

During import, Grafana asks you to map the data source placeholders (like ${DS_PROMETHEUS}) to real data sources and to choose a folder. If the uid already exists, you can choose to overwrite.

Transformations reshape query results in the browser before they reach the visualization. They are handy when you cannot, or do not want to, change the query itself.

• Merge and Join by field: combine results of several queries
• Organize fields: rename, reorder or hide columns
• Filter data by values: drop rows that do not match
• Group by and Reduce: aggregate rows or series
• Add field from calculation: derive a new column

They run in order, and since they happen client-side, very large results can make them slow.

Folders organize dashboards, library panels and alert rules. They are also the main place to set permissions, because everything inside a folder inherits the folder's access settings.

Nested folders are supported, so you can mirror team structure, for example Platform / Databases. Anything not in a folder goes into the General folder, which is best avoided in larger setups.

Grafana treats Prometheus as a data source. When a panel refreshes, Grafana's backend sends your PromQL expression to the Prometheus HTTP API (/api/v1/query_range) with a start, end and step, then plots the returned series.

The step is derived from the panel width, time range and min interval, so you rarely set it by hand. Use the legend format to label series, for example {{instance}}, and enable exemplars to jump from a metric point to a trace.

sum by (instance) (
  rate(node_cpu_seconds_total{mode!="idle"}[$__rate_interval])
)

Grafana only visualizes here. Scraping, storage and recording rules stay in Prometheus.

They solve different halves of monitoring and are usually deployed together.

Think of Prometheus as the data store and Grafana as the window onto it.

Both build dashboards, but they come from different ecosystems.

Many teams use Kibana for deep log search and Grafana for operational dashboards. Grafana also has an Elasticsearch data source if you want both views.

Grafana Alerting evaluates alert rules on a schedule, tracks the state of each resulting alert instance, and routes the ones that fire to people or systems.

1. An alert rule runs its queries and expressions (reduce, math, threshold) every evaluation interval
2. If the condition stays true for the pending period, the instance moves to Alerting
3. Firing instances go to the built-in Alertmanager
4. The notification policy tree matches their labels and picks a route
5. Alerts are grouped and sent to a contact point unless a silence or mute timing applies

flowchart LR
  A["Alert rule evaluation"] --> B{Condition true for pending period?}
  B -- No --> C["Normal / Pending"]
  B -- Yes --> D["Alerting instance"]
  D --> E["Notification policy matches labels"]
  E --> F{Silenced or muted?}
  F -- Yes --> G["No notification"]
  F -- No --> H["Contact point: Slack, email, PagerDuty"]

Each alert instance (one label set) is always in a state. The main ones are:

• Normal: the condition is not met
• Pending: the condition is met but the pending period has not elapsed yet
• Alerting: the condition has held long enough, so notifications are sent
• NoData: the query returned nothing
• Error: the query or evaluation failed

For NoData and Error you choose the behavior on the rule: raise an alert, treat as Normal, or keep the last state. If the pending period is 0, an instance skips Pending and goes straight to Alerting.

They answer two separate questions. A contact point says where and how to send a notification. A notification policy says which alerts go where, and how often.

So you create a contact point for a team once, then write policies like team=database that route to it.

Policies form a tree. The root is the default policy, and nested policies add label matchers such as severity=critical or team=database. An alert goes to the first matching branch, unless Continue matching is on.

Each policy also controls how notifications are batched:

Mute timings can silence a policy on a schedule, such as maintenance windows.

Where the rule is stored and evaluated is the key difference.

Data source-managed rules are a good fit when you already keep rules as code next to Prometheus or Mimir.

Grafana reads YAML files from /etc/grafana/provisioning at startup. There are subfolders for datasources, dashboards, alerting and plugins.

# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

# provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: default
    folder: Infra
    type: file
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

The dashboard provider watches the folder for JSON files. Provisioned dashboards cannot be saved from the UI unless allowUiUpdates is set, which keeps Git as the source of truth.

The official grafana/grafana Terraform provider lets you manage folders, data sources, dashboards, alert rules and contact points as code. You review changes in pull requests and apply them consistently across environments.

provider "grafana" {
  url  = "https://grafana.example.com"
  auth = var.grafana_token
}

resource "grafana_folder" "infra" {
  title = "Infra"
}

resource "grafana_dashboard" "node" {
  folder      = grafana_folder.infra.uid
  config_json = file("node.json")
}

Authenticate with a service account token rather than a personal login.

$__interval is calculated from the time range and panel width. Zoom in and it can shrink below your scrape interval, so rate() sees fewer than two samples in the window and returns nothing.

$__rate_interval fixes that. It is at least four times the scrape interval, calculated as max($__interval + scrape_interval, 4 * scrape_interval), which guarantees enough samples. Set the correct scrape interval in the Prometheus data source settings for this to work well.

# can show gaps when zoomed in
rate(http_requests_total[$__interval])

# safe
rate(http_requests_total[$__rate_interval])

Repeating lets one panel or row clone itself once per value of a multi-value variable. Pick three servers in the $server dropdown and you get three panels, each with $server locked to one value.

1. Create a variable with Multi-value or Include All enabled
2. Open the panel or row options and choose the variable under Repeat options
3. Pick the direction (horizontal or vertical) and the maximum panels per row

Repeated rows are useful when each server needs a full group of panels, while repeated panels suit a single chart per instance.

Both are enabled in grafana.ini. LDAP uses a separate ldap.toml to map directory groups to Grafana roles. OAuth providers such as GitHub, Google, GitLab and Azure AD have their own [auth.*] sections.

[auth.github]
enabled = true
client_id = YOUR_CLIENT_ID
client_secret = YOUR_CLIENT_SECRET
scopes = user:email,read:org
allowed_organizations = my-org
role_attribute_path = contains(groups[*], 'sre') && 'Editor' || 'Viewer'

The role mapping is the key part. Without it, every new user lands with the default role. SAML is also available in Enterprise and Cloud.

Legacy API keys were tied to an org with no real identity. They were hard to audit, could not easily be disabled, and many teams ended up with long-lived keys nobody owned.

A service account is a non-human identity with a role, and it can have several tokens. Tokens can expire, be revoked individually, and the account itself can be disabled. Actions are logged against the service account.

Grafana deprecated API keys and migrated existing ones to service accounts.

Run several Grafana instances behind a load balancer, all using the same external database, MySQL or PostgreSQL. The default SQLite file is local to one server and cannot be shared.

• Move sessions to the database or a remote cache such as Redis so users stay logged in on any node
• Keep identical configs, plugins and provisioning files on every instance
• Enable HA for alerting by setting ha_peers in [unified_alerting] so instances gossip and avoid duplicate notifications
• Make the database itself highly available

flowchart TD
  U[Users] --> LB["Load balancer"]
  LB --> G1["Grafana 1"]
  LB --> G2["Grafana 2"]
  G1 --> DB[(Shared MySQL or PostgreSQL)]
  G2 --> DB
  G1 <-->|alert gossip| G2

Start by finding what is slow. Open the Query inspector on a panel and compare query time with rendering time. Then work through these fixes:

1. Reduce the default time range and the number of panels on one dashboard
2. Lower Max data points or raise the min interval so fewer points come back
3. Use recording rules in Prometheus for expensive aggregations
4. Use instant queries for Stat and Gauge panels
5. Avoid heavy regex matchers and high-cardinality labels
6. Be careful with variables that refresh on every time range change
7. Enable query caching if you have Enterprise or Cloud
8. Split a huge dashboard into several focused ones

Panels below the fold load only when scrolled into view, so layout order matters too.

Work from the outside in, so you rule out simple causes first.

1. Check the time range and the dashboard timezone
2. Open the data source and run Save & test to confirm connectivity
3. Run the same query in Explore to see if it returns anything
4. Check variables: an empty value or an All regex can make the query match nothing
5. Confirm metric and label names exist in the backend
6. Look at the Query inspector response for errors or empty frames
7. Read the Grafana server log at /var/log/grafana/grafana.log

Permissions on the data source or folder can also hide results from some users.

Grafana is often exposed to many users, so tighten these defaults:

1. Change the default admin password, or disable the login form if you use SSO
2. Serve over HTTPS (protocol = https, with cert and key) or terminate TLS at a proxy
3. Disable anonymous access and user sign-up unless you need them
4. Set cookie_secure = true and a suitable cookie_samesite
5. Keep allow_embedding off unless required
6. Replace the default secret_key, which encrypts stored credentials
7. Give data sources least-privilege, read-only credentials
8. Use service accounts for automation and keep Grafana updated

Correlation works through shared identifiers and links between data sources. Three mechanisms matter most:

• Exemplars: Prometheus histogram samples can carry a trace ID that links straight to Tempo
• Derived fields: Loki extracts a trace ID from a log line and links to the trace
• Trace to logs and metrics: Tempo opens related logs or metrics for a span using shared labels like service name

flowchart LR
  M["Metric spike with exemplar"] --> T["Trace in Tempo"]
  L["Log line with traceID"] --> T
  T --> L2["Related logs in Loki"]
  T --> M2["Related metrics in Prometheus"]

Consistent labels across all three signals make the jumps accurate.

Alert annotations and notification messages use Go templating. Labels are available through $labels and query results through $values.

summary:     High CPU on {{ $labels.instance }}
description: CPU is at {{ printf "%.1f" $values.B.Value }}%

{{ define "slack.title" }}
  [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
{{ end }}

Named templates such as slack.title are defined under notification templates and then referenced from a contact point, so one template serves many integrations.

Tell Grafana its public URL and that it is served from a sub-path. Otherwise redirects and static assets point to the wrong place.

[server]
domain = example.com
root_url = %(protocol)s://%(domain)s/grafana/
serve_from_sub_path = true

location /grafana/ {
  proxy_pass http://localhost:3000/;
  proxy_set_header Host $http_host;
}

If you use Grafana Live, also forward websocket traffic with the Upgrade and Connection headers.

Loading a dashboard involves the browser, the Grafana backend and each data source. The backend acts as a secure query proxy, so credentials never reach the browser.

sequenceDiagram
  participant B as Browser
  participant G as Grafana backend
  participant D as Data source
  B->>G: GET /api/dashboards/uid/abc
  G-->>B: Dashboard JSON
  B->>G: POST /api/ds/query (per panel)
  G->>G: Check permissions, decrypt credentials
  G->>D: Native query (PromQL, SQL, LogQL)
  D-->>G: Result
  G-->>B: Data frames
  B->>B: Apply transformations and render

The dashboard JSON arrives first, then each panel posts its queries to /api/ds/query. The backend authorizes the request, loads the data source config, forwards a native query, and returns data frames. Transformations and rendering happen in the browser.

Use Mixed when a single panel needs queries from more than one data source. Each query row picks its own data source.

Typical cases are comparing production and staging metrics from two Prometheus servers, or putting a business number from PostgreSQL next to an infrastructure metric on the same graph. You can then use transformations to join or merge the results.

If you only need the same dashboard for different environments, a data source variable is cleaner, because it swaps the data source for the whole dashboard instead of mixing inside one panel.

The access mode decides who connects to the data source.

Choose Server mode for almost everything. Browser mode also requires CORS configuration on the data source.

Back up the pieces that hold state, not just dashboards:

1. The database: copy /var/lib/grafana/grafana.db for SQLite, or dump MySQL or PostgreSQL
2. grafana.ini and any ldap.toml
3. The provisioning folder
4. Custom plugins in /var/lib/grafana/plugins
5. The secret_key value

The secret key is easy to forget. Data source passwords in the database are encrypted with it, so a restore without the same key leaves those credentials unusable.

To restore, stop Grafana, put the database, config and plugins back, then start it. For extra safety, also keep dashboards as JSON in Git or Terraform.