Monitor Supabase with Grafana: dashboards and alerting
You have a Supabase project and want production-grade observability: a dashboard that answers “what is on fire” and alerts that fire before the answer matters. Since July 2026 there is a one-click Grafana Cloud integration in the Supabase Dashboard that provisions everything - authentication, metric scraping, and a pre-built dashboard over 200+ metrics - on every plan including free.1 This covers that path, the manual equivalents, how to read the dashboard, and the alert rules worth running. The alert rule set is the checkable form of the coverage decisions in the DR tiers reference.
Prerequisites: a Supabase project, and for the manual paths the project’s
legacy service_role JWT (Project Settings > API Keys > Legacy API keys).
No credit card on either side for the free tiers.
Constants
Section titled “Constants”| Thing | Value | Where it comes from |
|---|---|---|
| Metrics endpoint | https://YOUR_PROJECT_REF.supabase.co/customer/v1/privileged/metrics | The Supabase Metrics API, one endpoint per project |
| Auth | HTTP basic, username service_role, password = the legacy service_role JWT | Verified live 2026-08-18: the endpoint 401s on the new-format sb_secret_... keys (as basic-auth password AND as bearer token); the JWT is the working credential |
| Scrape interval | 30s on both manual paths (Part 2 and Part 3) | The Supabase docs recommend 1 minute because upstream refreshes about every 60s,2 but Grafana’s $__rate_interval is max($__interval + scrape_interval, 4 * scrape_interval) on the data source’s Scrape interval setting and resolves to 1m at common zooms, and a 1m window over 60s scrapes holds one sample, so rate() panels show no data (Part 3; reasoned from the Grafana docs, not reproduced). 30s guarantees two samples per window; the second scrape mostly re-reads the same values, which is cheap. The one-click path (Part 1) sets its own interval, which was not read |
| Metric volume | ~200 families, with per-CPU and per-device label fan-out | The full list is in the repo’s docs/metrics.md3 |
| Grafana Cloud free tier | 10k active series/mo, 14-day retention, 3 users | Enough for several projects’ worth of this endpoint4 |
| Grafana Cloud Pro | from $19/mo, 13-month metrics retention | When 14 days of trend stops being enough5 |
| Free-tier alert rules | 500 rules, 1000 alert instances per rule | Grafana-managed alert limit, Free Forever plan6 |
Which path
Section titled “Which path”| Path | Setup effort | Retention | Pick when |
|---|---|---|---|
| One-click Grafana Cloud | One button in the Supabase Dashboard | 14d free / 13mo paid | Default. Nothing to operate, dashboards pre-installed |
| Manual Grafana Cloud | Scrape job + dashboard import | Same | Custom scrape topology, existing stack, non-standard auth |
| Self-hosted Prometheus + Grafana | You run both | Yours to set | You already operate monitoring infra, want long retention or local alert routing |
Scale check before committing to a path: every project is its own scrape job with basic auth in scrape-config, so a federated account with six to ten projects means that many jobs and that many secret keys. The endpoint serves roughly 200 metric families (with per-CPU and per-device fan-out), which will very likely exceed the Grafana free tier’s 10k active series across that many projects - plan for paid or aggressive series pruning. The 14-day free retention is also shorter than the incident-to-renewal cycle you will want trend data for; anything you intend to argue with months later needs the long-retention column.
All three read the same Metrics API and can load the same dashboard - the
definitions in the supabase/supabase-grafana repo power the one-click
integration, the Grafana Cloud connection, and self-hosted imports
alike.1 One caution about that repo: it bills itself as an example,
not a production deployment.7 Treat its compose file as a reference,
not a stack to run as-is.
Part 1: one-click Grafana Cloud
Section titled “Part 1: one-click Grafana Cloud”- Open the project in the Supabase Dashboard, go to Integrations, select Grafana Cloud Observability Platform, click Install.
- Choose an existing Grafana Cloud workspace or create one - new accounts land on the free tier by default.
- Authorize the connection. Credentials are exchanged between Supabase and Grafana Cloud directly; no keys to copy.
- Metrics start flowing immediately, and the Supabase Project dashboard is pre-installed when you arrive in Grafana. A short tutorial points at the key panels, inviting teammates, and configuring alerts.8
Done. Skip to reading the dashboard.
Part 2: manual Grafana Cloud
Section titled “Part 2: manual Grafana Cloud”For an existing stack, a scrape topology the one-click path does not offer, or credential handling you want to control.2
- Sign in to Grafana Cloud, create or select a stack with Prometheus metrics enabled.
- Connections > Add new connection > Supabase. Provide the project ref,
the Metrics API endpoint, and the legacy
service_roleJWT as the basic-auth password (the new-formatsb_secret_...Secret API keys are NOT accepted by the endpoint as of 2026-08-18 - verified 401 against a live project). Name the scrape job after the environment (production-ap-southeast-1), not the project - the name is what alert labels carry. - Set the scrape interval to 30s (the Supabase docs say 1 minute; the
rate()reason for 30s is in Part 3 and applies to any Prometheus-backed panel, though the empty-panel symptom was reasoned from$__rate_intervalrather than reproduced on the managed agent), test the connection, save. Grafana Cloud deploys a managed agent that scrapes the endpoint and forwards to your hosted Prometheus. One scrape job per project - basic auth is per-scrape-config, so projects with different secrets cannot share a job. - Import the dashboard: Dashboards > New > Import, paste the raw
dashboard.json, and select the Prometheus datasource that receives the metrics. The Grafana Cloud Connections tile can also install it for you, along with a scrape-health overview dashboard.9
Part 3: self-hosted Prometheus
Section titled “Part 3: self-hosted Prometheus”The scrape config is short; the operational details around it (retention, firewall path, rule files) are the same as any exporter, and Add Prometheus monitoring to a Compose Postgres stack walks them for a compose host.
scrape_configs: - job_name: supabase-production metrics_path: /customer/v1/privileged/metrics scheme: https scrape_interval: 30s # see the rate-interval gotcha below basic_auth: username: service_role password_file: /run/secrets/sb_metrics_key # file holds the legacy service_role JWT static_configs: - targets: ['YOUR_PROJECT_REF.supabase.co:443']30s rather than the documented 1m, deliberately: Grafana’s
$__rate_interval resolves to 1m at common zooms per the
Grafana docs,
and a 1m left-open rate window over 60s scrapes holds at most one sample, so
rate() is undefined and panels render “No data”. 30s guarantees at least
two samples per window.
If you run more than a couple of projects, generate the jobs rather than
hand-editing: one job per project, each with its own 0600 credential file, so
no secret sits in prometheus.yml.
Multi-project: names in the dropdown, not refs
Section titled “Multi-project: names in the dropdown, not refs”Two small additions make a multi-project setup livable. First, give each job a static name label next to the ref label:
static_configs: - targets: ['YOUR_PROJECT_REF.supabase.co:443'] labels: supabase_project_ref: 'YOUR_PROJECT_REF' supabase_project_name: 'production' # what humans call itSecond, make the dashboard’s $project variable display the name while
filtering by the ref - query_result with a JS-syntax named-group regex
((?<text>...), NOT the RE2/Python (?P<text>...) form - the frontend
compiles the variable regex in JavaScript and rejects the latter with
“invalid regexp group”):
query_result(max by (supabase_project_name, supabase_project_ref) (node_load1{job=~"supabase-.*"}))regex: /supabase_project_name="(?<text>[^"]+)".*supabase_project_ref="(?<value>[^"]+)"/And if the dashboard filters on a SECOND variable (supabase_identifier in
the stock dashboard), cascade it off the project or the two filters disagree
and every panel goes empty:
label_values(node_load1{supabase_project_ref="$project"}, supabase_identifier)Both variables are then fully dynamic - adding a project is one scrape job, zero dashboard edits.
Part 4: reading the dashboard
Section titled “Part 4: reading the dashboard”The integration installs two dashboards.9 Metrics endpoint scrape overview is scrape health - up/down, duration, samples. Open it when data looks stale, not before. Supabase Project is the operational one: a row of headline stats, then collapsible rows of panels. The 200+ number counts every panel; the ones that answer production questions are a small subset.
Headline stats across the top: CPU Busy, Sys Load (5m/15m), RAM Used, Root FS Used, SWAP Total, Data Disk Total, Uptime. Then:
| Question | Panels (row) |
|---|---|
| Is the database up? | Postgres status, DB Mode, In Recovery (Postgres) |
| CPU or memory pressure? | CPU Basic, Memory Basic (Basic CPU / Mem / Net / Disk); System Load (System Misc); major faults, swap, OOM Killer (Memory Vmstat) |
| Disk filling or slow? | Disk Space Used, EBS IO Balance (Basic row); Disk Average Wait Time, Average Queue Size (Storage Disk); Filesystem space available (Storage Filesystem) |
| Connection pressure? | Client connections, pgbouncer status (Postgres) |
| Replication falling behind? | Realtime replication status / lag (Postgres: realtime) |
| Checkpoint churn? | bgwriter stats: checkpoints - the requested-vs-timed mix (Postgres: bgwriter) |
| Growth and query shape? | Database size, Query stats, pg stats, Conflicts (Postgres) |
The remaining rows - Memory Meminfo, System Timesync, System Processes,
Systemd, Network Sockstat/Netstat, Node Exporter - are node-exporter deep
cuts. Leave them collapsed until a headline panel points at one. Expect many
of them to be empty permanently: the endpoint serves a CURATED subset of
node_exporter, not the full collector set. Measured 2026-08-18 against a live
free-tier project: 87 of the 148 panels have data; the empty 60 are mostly
those deep cuts (sockstat, netstat, timex, softnet, entropy, systemd,
conntrack, interface speed), plus panels that depend on project features -
the Supavisor row only has data on projects that run Supavisor (pgbouncer
projects stay empty there), and the realtime replication panels need realtime
replication in use. Empty there means “not served”, not “broken scrape”.
The Uptime stat is the one visible casualty in the stock headline row:
node_boot_time_seconds/node_time_seconds are not served, so it shows N/A
- swap the query for
time() - process_start_time_seconds{service_type="postgresql"}(postgres process uptime as the compute-uptime proxy) if you want it back.
Two habits make the dashboard pay off:
- Read rates over windows, not point values. A MemAvailable snapshot can
look healthy while the working set pages in from disk all hour - the signal
is the
node_vmstat_pgmajfault/node_vmstat_pswpinrate over a window, not the gauge. - The requested/timed checkpoint mix is the write-pressure tell. Timed
checkpoints are the healthy interval-driven case; a rising requested share
means WAL fills before
checkpoint_timeoutand each checkpoint is a burst of full-page writes.
Part 5: alerting
Section titled “Part 5: alerting”Supabase serves the metrics but has no in-product alert notification path - alerting is on the Grafana/Prometheus side. Two rule sources, then how to wire them in Grafana Cloud.
The rule set
Section titled “The rule set”Eighteen rules in two groups: twelve run on the Metrics API alone, one
(SupabaseConnectionCeiling) needs a connection-count family the endpoint
was not confirmed to serve, and five need a scrape source you add yourself. Thresholds are starting values tuned
for tier-scaled managed Postgres, not universal constants. Three of the
thirteen originate in the repo’s docs/example-alerts.md10;
the rest come from pg-analyser’s threshold catalogue.
The core thirteen - the endpoint covers twelve of these; the thirteenth is discussed under the YAML. Straight from the metric families the Metrics API serves:
| Alert | Condition | Severity |
|---|---|---|
PostgresDatabaseDown | pg_up == 0 for 5m | critical |
SupabaseOomKill | any node_vmstat_oom_kill rate over 1h | high |
SupabaseCpuSaturated | >= 50% of 1h window at/above 80% CPU | high |
SupabaseWalArchivalBacklog | pending WAL archival >= 1, mean over 1h | high |
SupabaseMemorySaturated | >= 30% of 1h window at/above 85% memory | medium |
SupabaseDiskFull | data disk >= 80% used, held 10m | medium |
SupabaseMajorPageFaults | major faults >= 20/s, mean over 1h | medium |
SupabaseSwapIn | swap-in >= 2 pages/s, mean over 1h | medium |
SupabaseCheckpointPressure | requested / (requested + timed) checkpoints >= 0.3, each leg window-smoothed | medium |
PostgresReplicationLagHigh | lag > 600s and still rising over 10m | warning |
PostgresDatabaseSizeGrowth | size up > 20% over 12h | warning |
SupabaseScrapeDown | up < 1 for 10m - the scrape itself is failing | high |
SupabaseConnectionCeiling | backends / max_connections_connection_count >= 80% for 10m | medium |
SupabaseScrapeDown is the rule that covers a dead scrape: when the scrape
goes down (a rotated credential is the usual cause), every other rule goes
no-data at once, and this one still fires. SupabaseConnectionCeiling was
excluded from the rule set for a long time because the configured limit was
not on the scrape - the endpoint now exports it as max_connections_connection_count
(verified 2026-08-18), which is what makes the rule possible.
PostgresReplicationLagHigh needs two footnotes. The metric family
(physical_replication_lag_...) measures the PLATFORM’s own physical
replication - it says nothing about a hand-rolled logical standby built
with subscriptions, which needs its own canary-insert check instead
(exactly what the runbook’s
probe_canary table is for; its Part 4 cutover rehearsal verifies the
standby through it). And
600s is a starting value, not a sensible default for a healthy link: a
measured cross-region logical standby sat at 245-276ms of lag, so a rule
three orders of magnitude above steady state fires only when the link is
effectively dead. Tighten toward your own baseline - the right threshold
is “unusual for this account”, not a round number.
The optional five - each needs an extra scrape source. PSI stall rules need a node_exporter with the pressure-stall collector; the EBS burst-balance rules need a CloudWatch exporter. On the Supabase endpoint alone they stay silent, so only run them with the source they name - an inert rule reads as coverage it is not.
| Alert | Condition | Needs |
|---|---|---|
SupabasePsiCpuStall / SupabasePsiMemoryStall / SupabasePsiIoStall | PSI stall >= 20% over 1h | node_exporter pressure collector |
SupabaseEbsIopsBalanceLow / SupabaseEbsThroughputBalanceLow | EBS burst balance <= 20%, held 10m | cloudwatch_exporter |
The EBS pair maps to the Supabase Disk IO Budget: Nano through Medium run on the gp3 baseline (3,000 IOPS / 125 MB/s) plus burst credits and cannot provision extra IOPS - a depleting balance is the early warning before throttling clamps throughput to baseline.
When the rule set helps, and when it cannot
Section titled “When the rule set helps, and when it cannot”The rule set’s vantage is INSIDE the project, watching the project’s own processes. That fixes what it can see and, by construction, what it cannot:
It helps when the mechanism is resource or process state. Disk full, OOM, CPU saturation, checkpoint pressure, replication lag, WAL backlog, runaway database growth - these live exactly where the metrics live, and the rule set catches them early. This is most capacity incidents and every slow-degradation story.
It cannot help when the mechanism is anywhere else. The process is
healthy and the answer is wrong - that is a whole genre, and it shows
pg_up == 1 with a green dashboard all the way through:
- Request-outcome failures: PostgREST rejecting valid tokens, Auth failing logins, REST 5xx on one table, RLS answering empty. The Metrics API has no per-request signal at all.
- Platform-layer incidents: the shared gateway in front of your project (502/520-class), DNS, TLS, control-plane unavailability - none of it is in your project’s process state.
- Billing restrictions: an org hitting the Fair Use wall answers 402 on every API request while every process reports healthy.
- Anything client-side or between: regional network partitions, client bugs, a bad deploy of YOUR app. Your scrape target is fine; your users still can’t reach it.
Coverage is a property of vantage points, not of rule count - so pair the rule set with a synthetic probe that exercises the real surfaces from OUTSIDE, with a real token (the probe table in the resilience runbook Part 1 is exactly that), and treat the billing/status-page signals as their own channel. The dashboard’s System Timesync panel is adjacent to one of these failure modes but ships collapsed with no rule attached.
Three integrity notes on specific rules:
SupabaseWalArchivalBacklogis the verification behind PITR’s two-minute worst-case RPO: that number is only true while WAL keeps shipping on its two-minute interval. If you sell yourself the PITR tier, this alert is the only thing that tells you the RPO stopped being true. See the DR tiers reference.SupabaseCheckpointPressuresurvives Postgres 17: upstream moved the checkpoint counters topg_stat_checkpointer, but the Supabase endpoint still servespg_stat_bgwriter_checkpoints_req_totaland_timed_totalon PG17 projects (verified 2026-08-17 against a live PG17.6 project - the rule is not inert). The general warning stands, though: metric families can shift across Postgres versions, so a rule that has never fired is worth one manual query against the endpoint before you trust it.SupabaseCheckpointPressureneeds each ratio leg window-averaged BEFORE dividing. Checkpoint counters move in impulses - one timed checkpoint percheckpoint_timeout- so a barerate(...[5m])share sits at 0 or 1 at almost every evaluation, and one forced checkpoint flips it over the line for a single interval: a firing/resolved flap on a healthy database. Measured live (2026-08-18): 0.4% requested share over 7 days while the unsmoothed rule was flapping. The smoothed form averages eachrate()over the window first, so the share is the window’s requested proportion and only sustained WAL pressure holds it past 0.3.
The rules as YAML
Section titled “The rules as YAML”Prometheus rule-file form, genericised (no project ref pinned - scope by
job or supabase_project_ref in multi-project setups). The sustained-fraction
rules count the share of window samples past the line rather than requiring an
unbroken run, which is what “sustained” means on a spiky metric; for: would
answer a different question. The checkpoint ratio averages each leg over the
window before dividing, for the impulse-counter reason in the integrity note
above.
groups: - name: supabase-core rules: - alert: PostgresDatabaseDown expr: pg_up == 0 for: 5m labels: {severity: critical} annotations: summary: PostgreSQL database is down
- alert: SupabaseScrapeDown expr: up{job=~"supabase-.*"} < 1 for: 10m labels: {severity: high} annotations: summary: Metrics scrape failing - every other rule is silent, check the credential first
- alert: SupabaseCpuSaturated expr: >- (count_over_time((clamp_min(100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100, 0) >= 80)[1h:5m]) / count_over_time(clamp_min(100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100, 0)[1h:5m])) >= 0.5 labels: {severity: high} annotations: summary: CPU sustained high - queries are queueing for cores
- alert: SupabaseMemorySaturated expr: >- (count_over_time(((1 - avg(node_memory_MemAvailable_bytes) / avg(node_memory_MemTotal_bytes)) * 100 >= 85)[1h:5m]) / count_over_time(((1 - avg(node_memory_MemAvailable_bytes) / avg(node_memory_MemTotal_bytes)) * 100)[1h:5m])) >= 0.3 labels: {severity: medium} annotations: summary: Memory sustained near the ceiling
- alert: SupabaseDiskFull expr: >- 100 - (avg(node_filesystem_avail_bytes{mountpoint="/data"}) * 100 / avg(node_filesystem_size_bytes{mountpoint="/data"})) >= 80 for: 10m labels: {severity: medium} annotations: summary: Data disk past the fill threshold - a full disk forces Postgres read-only
- alert: SupabaseOomKill expr: avg_over_time(sum(rate(node_vmstat_oom_kill[5m]))[1h:5m]) > 0 labels: {severity: high} annotations: summary: Kernel OOM killer fired - even one event means requests were killed
- alert: SupabaseMajorPageFaults expr: avg_over_time(sum(rate(node_vmstat_pgmajfault[5m]))[1h:5m]) >= 20 labels: {severity: medium} annotations: summary: Working set paging in from disk (major faults)
- alert: SupabaseSwapIn expr: avg_over_time(sum(rate(node_vmstat_pswpin[5m]))[1h:5m]) >= 2 labels: {severity: medium} annotations: summary: Working set paging in from swap
- alert: SupabaseCheckpointPressure expr: >- avg_over_time((sum(rate(pg_stat_bgwriter_checkpoints_req_total[5m])))[1h:5m]) / (avg_over_time((sum(rate(pg_stat_bgwriter_checkpoints_req_total[5m])))[1h:5m]) + avg_over_time((sum(rate(pg_stat_bgwriter_checkpoints_timed_total[5m])))[1h:5m])) >= 0.3 labels: {severity: medium} annotations: summary: Most checkpoints forced by WAL filling - raise max_wal_size
- alert: SupabaseWalArchivalBacklog expr: avg_over_time(max(pg_ls_archive_statusdir_wal_pending_count)[1h:5m]) >= 1 labels: {severity: high} annotations: summary: WAL archival falling behind - PITR and disk headroom at risk
- alert: PostgresReplicationLagHigh expr: >- (physical_replication_lag_physical_replication_lag_seconds > 600) and (rate(physical_replication_lag_physical_replication_lag_seconds[10m]) > 0) for: 5m labels: {severity: warning} annotations: summary: Replication lag over 10m and growing
- alert: PostgresDatabaseSizeGrowth expr: >- (pg_database_size_mb - pg_database_size_mb offset 12h) / pg_database_size_mb offset 12h * 100 > 20 for: 5m labels: {severity: warning} annotations: summary: Database grew more than 20% in 12hSupabaseConnectionCeiling is in the table and not in the block. The
denominator, max_connections_connection_count, was verified on the endpoint
(2026-08-18), but the backend-count family to divide by was not recorded in
that check, and the repo’s metric list names no Postgres backend count either
(read 2026-09-03). Query the endpoint once for the connection-count family it
serves before writing the rule, and preview it against real data; a guessed
numerator is an inert rule.
Every rule here pairs with a remediation: size up the compute tier, raise
max_wal_size, cut backend count through the pooler, add indexes to shrink
the hot set. The pg-analyser project generates this same rule set with full
remediation and runbook annotations from its threshold catalogue
(alerts-init), which is how the YAML above stays in sync with a monitoring
report instead of drifting from it.
What not to alert on
Section titled “What not to alert on”Some obvious candidates are traps:
- Cache hit ratio - a rate-window rule pages on idle projects; the ratio only means anything above a block-volume floor that a range window cannot express.
- Disk fill projection -
predict_linearextrapolates an auto-expansion as a cliff; the series has to be segmented on resize events first, which PromQL cannot do. - Connection ceiling % - was a trap when the denominator was not
exported; as of 2026-08-18 the endpoint serves the configured limit as
max_connections_connection_count(verified live: 60 on free-tier projects), so a ceiling line or ratio IS now possible - and the rule set above carries it (SupabaseConnectionCeiling). Note the name - it is NOTpg_settings_max_connections. - Deadlocks - the counter is cumulative since stats reset; a threshold on it fires forever after the first event.
A rule set that alerts on what Prometheus cannot see is worse than no rule set.
Wiring alerts self-hosted (Alertmanager)
Section titled “Wiring alerts self-hosted (Alertmanager)”On the self-hosted path the wiring is: Prometheus alerting: block pointing
at an Alertmanager, alerts.yml mounted as a rule_files: entry, and a
receiver for delivery. Any email API with an SMTP relay works as the receiver
- with Resend for example: smarthost
smtp.resend.com:587(STARTTLS; 465/2465 are implicit TLS, 25/587/2587 STARTTLS), username literallyresend, password = the API key,smtp_fromon a domain verified in the Resend account. Group bysupabase_project_ref, or alerts from different projects coalesce into one notification. And verify end-to-end with a synthetic alert POSTed straight to Alertmanager’s/api/v2/alerts- that exercises the whole chain except the rule engine.
Scoping note for multi-project rule files: if the same Prometheus also scrapes
other node/postgres exporters (edge routers, other databases), an unscoped
rule file fires the Supabase node_* rules on THOSE hosts too. Require the
project label instead: generate with a label-present matcher
(supabase_project_ref=~".+" - pg-analyser’s alerts-init --ref '~.+' emits
exactly this) so the rules cover every project job and nothing else.
Wiring alerts in Grafana Cloud
Section titled “Wiring alerts in Grafana Cloud”New Grafana Cloud stacks default to Grafana-managed alerting; data source-managed rules are deprecated there.6 That means you create rules in the UI (or via the provisioning API / Terraform), not by mounting a rule file:
- Alerting > Alert rules > New alert rule. Pick the Prometheus datasource, paste the query in Code mode, set the threshold condition, and Preview - the preview evaluates the expression against real data, so a typo’d metric name shows up as empty before the rule ever ships.
- Set the evaluation group interval (1m; two 30s scrapes per evaluation) and a pending period. For the sustained-fraction rules the window is already in the expression; a short pending period (or none) is right. For gauge rules like disk fill, hold 10m so one bad scrape cannot page.
- Set no data handling explicitly:
pg_up == 0already covers the database being down, but if the scrape itself dies, every rule goes no-data at once. Alerting on scrape absence (the scrape-overview dashboard’s up metric, or no-data = alerting on one dedicated rule) is what catches a rotated key or an expired token. - Add annotations a responder can act on: what the signal means, the fix, and a runbook link. A page that says “CPU high” and nothing else costs the responder the minutes the annotation would have saved.
- Contact points (Alerting > Contact points) carry the notification:
email, Slack, PagerDuty, webhook. Notification policies route by label -
the
severitylabels above are the routing key, socriticalcan page whilewarninglands in a channel.
Evidence
Section titled “Evidence”| Claim | How it was checked | Tested vs documented |
|---|---|---|
| 87 of the 148 panels have data (2026-08-18) | Queried the live dashboard against a live free-tier project | Tested |
| 0.4% requested share over 7 days while the unsmoothed rule flapped (2026-08-18) | Measured live against the endpoint’s checkpoint counters | Tested |
max_connections_connection_count served, 60 on free-tier projects (2026-08-18) | Queried the live Metrics API endpoint | Tested |
| Cross-region logical standby lag at 245-276ms | Measured against a live cross-region logical standby | Tested |
401 for sb_secret_... keys, legacy service_role JWT accepted (2026-08-18) | Live requests against the endpoint, as basic-auth password and as bearer token | Tested |
pg_stat_bgwriter_checkpoints_req_total and _timed_total still served on PG17 (2026-08-17) | Queried a live PG17.6 project | Tested |
What to do about it
Section titled “What to do about it”| Practice | Rests on |
|---|---|
| Scrape at 30s on both manual paths (Part 2 and Part 3); do not follow the 1-minute figure from the Supabase docs. The one-click path (Part 1) sets its own interval, which was not read | Part 3: $__rate_interval resolves to 1m and a 1m window over 60s scrapes holds one sample. Reasoned from the Grafana docs; the empty-panel symptom on Grafana Cloud’s managed agent was not reproduced |
Until an sb_secret_... key is re-tested against the endpoint, keep the legacy API keys enabled while the scrape holds the service_role JWT | On 2026-08-18 the endpoint answered 401 to sb_secret_... keys as basic-auth password and as bearer token and accepted only the legacy service_role JWT; the Supabase Metrics API page read 2026-09-03 documents sb_secret_... as the basic-auth password,11 so re-test with one before treating the JWT as the only credential |
Run SupabaseScrapeDown (up{job=~"supabase-.*"} < 1 for 10m), or set no-data handling to alerting on one dedicated rule | Part 5: every rule goes no-data at once when the scrape dies, and a rotated credential is the usual cause |
Before writing SupabaseConnectionCeiling, query the endpoint for the connection-count family it serves and preview the rule against real data | Evidence row: max_connections_connection_count served, 60 on free-tier projects (2026-08-18); the numerator was not recorded |
Count active series per job (count({job="supabase-production"})) before picking a Grafana Cloud tier, and drop unused families with metric_relabel_configs if the total nears 10k | Constants: ~200 families with per-CPU and per-device fan-out against a 10k-series free tier. Design step; the per-project series count from the 2026-08-18 scrape was not recorded |
Scope every rule with supabase_project_ref=~".+" (or job=~"supabase-.*") when the same Prometheus scrapes other hosts | Scoping note under Alertmanager: an unscoped rule file fires the node_* rules on every scraped host. Reasoned, not measured |
Verification
Section titled “Verification”| Check | Where | Expect |
|---|---|---|
| Scrape healthy | Metrics endpoint scrape overview dashboard, or up in the expression browser | 1, scrape duration well under the interval |
| Panels render at any zoom | Supabase Project dashboard at 1h and 24h ranges | Data in rate() panels, not “No data” |
| Alert expressions valid | Rule editor Preview per rule | Series return, threshold line visible |
| Notifications deliver | Contact points > Test | Message lands in the channel |
| Rules loaded (self-hosted) | curl -s localhost:9090/api/v1/rules | jq '.data.groups[] | {name, rules: (.rules|length)}' | Every group and rule listed |
| Rule YAML valid (self-hosted) | promtool check rules alerts.yml (ships in the prom/prometheus image) | SUCCESS |
Gotchas and lessons learned
Section titled “Gotchas and lessons learned”60s scrapes silently break rate() panels. Covered in Part 3 - the
single most common “the dashboard is empty” cause on self-hosted setups. Scrape
at 30s and move on.
The Metrics API is point-in-time; there is no history before you start scraping. Trends exist only from the moment a scraper runs, and on the free tier they age out after 14 days. If you need month-scale trends for capacity calls, that decides Pro vs self-hosted for you.
A 401 is a key problem, not a network problem. Check which credential you
handed the scrape first: verified 2026-08-18, the endpoint accepts the legacy
service_role JWT and returns 401 for the new-format sb_secret_... Secret
API keys - as basic-auth password and as bearer token alike. The Supabase
Metrics API page read 2026-09-03 documents sb_secret_... as the basic-auth
password,11 so re-test with one before treating the JWT as
the only credential; once an sb_secret_ key passes that re-test, prefer a
dedicated one for the scraper (it rotates independently and the JWTs are on
the deprecation path). Until then, keep the legacy API keys enabled while the
scrape holds the service_role JWT. The JWT is one of them, and the endpoint
accepted nothing else on 2026-08-18. Whether the Dashboard’s disable switch
401s the endpoint at once was not itself tested; what was tested is that no
other credential works. Re-test the endpoint with the scrape’s credential
after any key change.
The one-click path hides the scrape entirely. Convenient until you need to change the interval or relabel - then you are in the manual path anyway, and the manual scrape config above is the whole thing.
Alert on the scrape, not just the database. Every rule here goes quiet simultaneously when the scrape breaks. One rule on scrape presence (or no-data handling set to alerting) turns “monitoring silently dead” into a page.
Backfilling a counter into a series that already has live data manufactures
counter resets. If the backfill source’s baseline differs from the live
counter’s - a statistics database that started accumulating later than the
sensor’s own counter - every backfilled point below the live samples reads as
a reset-to-zero, and increase() adds the full counter value at each one.
Eight hourly points of about 20 kWh each came out as about 159 phantom kWh
here. resets(metric[3d]) is the smoke signal, but coarse query_range
steps hide the decreases; diff consecutive raw samples from an instant range
selector (metric[3d]). Fix by point-deleting the backfill timestamps:
PUT /api/v1/admin/tsdb/delete_series with start=end=<ts>. Backfill lands
on whole seconds and live scrape samples carry millisecond precision, so the
point delete touches no live data. And check first whether the counter even
has a gap - a counter with continuous live data needs no fill, because
increase() interpolates across a gap.
A raw-counter “all-time” panel lies after the first sensor rebase.
Utility sensors get recreated and the cumulative state starts a new epoch; a
panel plotting the raw counter carries every epoch step as a vertical cliff
forever. Plot cumulative-in-range instead: increase(counter[$__interval])
with a 1m min interval, plus an “Add field from calculation” transform. On
Grafana 13 the transform schema is mode: "cumulativeFunctions" with
cumulative: {reducer: "sum"} and replaceFields: true (keeps Time plus the
new field; with no field set it cumulates the first numeric field). The
pre-13 form (mode: "cumulative", reduce.fields) is not rejected - it
silently no-ops, and a downstream name filter then renders “Data is missing a
number field”. One more naming trap: a Prometheus legendFormat sets the
field’s display name (displayNameFromDS), while name-based transform
selectors match the field name, which stays Value.12
File reference
Section titled “File reference”| File | Where | What |
|---|---|---|
grafana/dashboard.json | supabase/supabase-grafana repo | the 200+ panel dashboard, imported everywhere |
docs/example-alerts.md | same | the three upstream starter rules |
docs/metrics.md | same | the full exported metric list |
prometheus.yml scrape config | yours | Part 3, one job per project |
alerts.yml | yours | the rule set above, mounted as a rule_files: entry |
References
Section titled “References”-
Supabase, “Observability for every Supabase project with Grafana Cloud,” Supabase Blog. https://supabase.com/blog/observability-for-every-supabase-project-with-grafana-cloud ↩ ↩2
-
Supabase, “Metrics API with Grafana Cloud,” Supabase Docs. https://supabase.com/docs/guides/monitoring-and-debugging/metrics/grafana-cloud ↩ ↩2
-
Supabase, “Exported metrics,” supabase-grafana repo. https://github.com/supabase/supabase-grafana/blob/main/docs/metrics.md ↩
-
Grafana Labs, “What’s Included in Grafana Cloud Free.” https://grafana.com/products/cloud/free-tier/ ↩
-
Grafana Labs, “Grafana Pricing.” https://grafana.com/pricing/ ↩
-
Grafana Labs, “Configure Grafana-managed alert rules,” Grafana Docs. https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/ ↩ ↩2
-
Supabase, “supabase-grafana README,” GitHub. https://github.com/supabase/supabase-grafana ↩
-
Grafana Labs, “How to monitor your Supabase projects: connect Grafana Cloud in one click,” Grafana Blog. https://grafana.com/blog/grafana-cloud-supabase-one-click-integration/ ↩
-
Grafana Labs, “Supabase integration for Grafana Cloud,” Grafana Cloud Docs. https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-infrastructure/integrations/integration-reference/integration-supabase/ ↩ ↩2
-
Supabase, “Example Alerts,” supabase-grafana repo. https://github.com/supabase/supabase-grafana/blob/main/docs/example-alerts.md ↩
-
Supabase, “Metrics,” Supabase Docs. https://supabase.com/docs/guides/telemetry/metrics ↩ ↩2
-
Grafana Labs, “calculateField.ts,” grafana repo, tag v13.1.3. https://github.com/grafana/grafana/blob/v13.1.3/packages/grafana-data/src/transformations/transformers/calculateField.ts ↩