SupabaseOperations and accessguide
Supabase resilience runbook: probes, TTL levers, edge cache, warm standby
Build the four layers of client-side resilience against Supabase platform incidents: detection (probes that see real outages), exposure reduction (token TTL), read survival (edge cache), and full HA (warm standby with cutover). The concepts and the evidence for every number are in the companion reference, Supabase incidents: what a client can actually do.
Prerequisites. A Supabase Management API personal access token, a
project on any paid plan (spend cap behaviour differs on Pro), bun or
Node, wrangler for the edge layer, pg_dump/psql for cold DR. The
standby layer needs a second project (any region, at the compute size you
will cut over to - see Part 4) - everything else uses the one you have.
Constants preamble
Section titled “Constants preamble”Measured facts every later step depends on (supabase-lab, 2026-08-15/16):
| Fact | Value |
|---|---|
| PostgREST iat/exp skew tolerance | ~30s (+30s passes, +31s rejects PGRST303) |
| jwt_exp config acceptance to issuer effect | ~6.5s |
| TPA resolution time | ~60-122ms |
| JWKS trust lag after TPA registration | ~30s cold, ~300ms warm |
| Standby initial sync (small table) | ~3.1-6.5s |
| Standby initial sync (1M rows) | 22.7s / 12.5s (two runs); lag ~245-276ms after |
| Standby replication lag (cross-region) | 34ms-1057ms |
| auth./storage. replication | zero changes at any tested size - do not plan on it |
| DDL on the primary while subscribed | stalls all table replication; standby-side DDL resumes in ~6.1s |
| Edge failover trip condition | 5xx, or the CF-wrapped 403 of an unroutable origin - never >=500 alone |
| Edge function wall clock | 150s idle -> 504 IDLE_TIMEOUT |
| Edge function cold start | ~1.4s first-after-idle only; then 121-302ms vs 98ms warm |
| Edge function CPU or memory exhaustion | 546 WORKER_RESOURCE_LIMIT, identical body for both (3s busy loop; 400 MB allocation) |
| Edge function deploy ceilings | 8 MB refused 413 request entity too large via API, lands via local CLI bundling; parallel deploys can return 201 and not land - one ceiling at a time |
| statement_timeout signature | verbatim 57014, ~3.5s wall (2s setting vs 5s sleep) |
| lock_timeout signature | verbatim 55P03, ~4.5s wall (3s setting) |
| pg_cron across a restart | resumes on schedule (2 -> 5 heartbeat rows), no doubling; the instant inside the restart is skipped and a run in flight fails (CB04/CB05) |
| Cold DR floor (10k rows via pooler) | dump 12.4s, restore 6.4s |
| Gateway behaviour on every response | sets Set-Cookie: __cf_bm |
Architecture overview
Section titled “Architecture overview”- Client -> Edge Worker (cache-first, cutover routing).
- Edge Worker -> Primary project normally, and -> Standby project (dashed) after cutover.
- Primary project -> Standby project over logical replication, WAL on the direct host.
- Synthetic probes (per-service, authed) -> Primary project.
Part 1: Detection - probes that see real outages
Section titled “Part 1: Detection - probes that see real outages”Anonymous network probes miss real outages. ACTIVE_HEALTHY means the
HTTP tier is up and says nothing about whether PostgREST can reach
Postgres, and TCP accept is not readiness, so every probe here is
authenticated and hits a real table.
- Create a canary table on each project you monitor:
curl -s -X POST \ -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ "https://api.supabase.com/v1/projects/$REF/database/query" \ -d '{"query":"create table if not exists public.probe_canary(id int primary key); insert into public.probe_canary values (1) on conflict do nothing;"}'- Probe per service, and record the PGRST code, not just the status:
# Data API - 200 with rows = healthy; capture body code on failurecurl -s -w '\n%{http_code}' \ -H "apikey: $PUBLISHABLE_KEY" -H "Authorization: Bearer $PUBLISHABLE_KEY" \ "https://$REF.supabase.co/rest/v1/probe_canary?select=id"The other three services need their own probes, each against a real operation rather than a health endpoint:
- Auth: admin-create a probe user and take the password grant (the
Part 2 commands). Probing an authenticated operation is a design
choice; neither lab run probed that path. Both sampled
GET /auth/v1/healthinstead: platform-downtime D01 (2026-08-04, 500 ms sampling, anon key) sawHTTP 521for 75 s across a restart, and compute-disk D09 (2026-10-23, 250 ms, anon key) saw it down 152.375 to 254.869 s across four resizes, within 1 s of a row read through REST in each. D09’s earlier 0 s (2026-08-19) is withdrawn: that probe sent no apikey, and the endpoint answered401without one on 2026-10-23. - Storage: fetch a real object. On a fresh project Storage lags
ACTIVE_HEALTHY, answeringTenantNotFoundand then429 SlowDownfor the first minutes (edge-resilience W21, 2026-08-17) - retry rather than page. - Pooler:
select 1over 6543 with a connect timeout. The error string names the operation underway (rows below). With the lab’s 5 s probe timeout, recovery in timeout mode resolved only to about 5.5 s; keep the timeout short if you want the window to the second. A probe that only connects does not see pool exhaustion: the client is accepted in 43 to 52 ms and the error arrives on its first statement (PC01), so run theselect 1. If a probe or the app retries a statement that failed on a reset connection, pause 300 ms first and not 0 ms. Through a toxiproxy TCP fault between the client and the 6543 pooler (2026-10-10, PC03), a retry-once at 0 ms rescued 0 of 30 failures after a 150 ms flap on node-postgres and on Prisma, and a retry at 300 ms rescued 25 of 25 and 30 of 30. The pause covers a 150 ms flap. After a 2 s reset or cut, 300 ms rescued 0 of 120 in each case on node-postgres and 30 and 10 of 120 on Prisma. postgres.js rescued every failure it surfaced at both delays. No first attempt failed after the fault cleared in the 111 runs that did not crash, so the three driver pools evicted dead connections themselves, but a node-postgresPoolwithout an'error'listener exited in 9 of 12 runs at the cut (Unhandled 'error' event).
Declare recovery only after sustained success (the lab required 5 s of passing samples), because the pooler queues before it refuses and one passing sample mid-outage reads as recovery; and void any probe row that was already failing before the operation started, or a bad credential becomes a platform outage (platform-downtime “Reproducing”).
- Alert on the code classes; each maps to one incident type:
| Signal | Meaning | Action |
|---|---|---|
| PGRST303 rate | JWT claim rejection (skew incident) | TTL lever (Part 2), wait for platform fix |
| PGRST301 | unknown signing key | key rotation/TPA config, not an outage |
| PGRST002 | schema-cache wedge, or a restart or restore in progress (a Data API read answered 503 PGRST002 for 14 s to 544 s through PITR restores (RP06/RP07), and as one of four failure modes during restarts (LO05, 2026-10-23)) | if no operation is running, select pg_notify('pgrst','reload schema'); during one, wait |
| PGRST000 / 5xx on a real table | DB or tier failure | incident response |
| 520/521/522 from CF front | gateway path | retry with backoff; cache serves reads. supabase-js GET and HEAD retry on 520 and 503 only (4 attempts at 1, 2, 4 s); 521, 522, 525 and 500 were each sent once (CR01a4, 2.112.3, injected statuses on a local fault proxy, 2026-10-10); a 503 with Retry-After: 2 was waited 2 s, and a real Data-API-off 503 with Retry-After: 0 was retried with no pause (CR01a6, CR01e1) |
| 402 | billing restriction (Fair Use Policy: quota, spend cap, overdue invoice) | fix billing - card, cap toggle, or usage - not the platform; details in the DR tiers reference |
pooler Failed to connect to database: {:error, :timeout} | project restart underway (158 s measured, platform-downtime D01, 2026-08-04; LO05 re-check, n=5, 2026-10-10: 50-58 s p50 per path); the 2026-10-23 restarts (D01, and LO05 at 145-151 s p50 per path) answered terminating connection due to administrator command instead, so the string does not name the operation by itself | wait and retry on every path; REST reads and Realtime subscriptions fail through it too |
pooler Failed to connect to database: {:error, :econnrefused} | compute resize up underway (207 s measured, D03, 2026-08-04); the 2026-10-23 resize-ups (115 to 218 s on the longest path, Micro and Medium) answered terminating connection due to administrator command instead, so the string does not name the operation by itself | wait and retry on every path; REST reads and Realtime subscriptions fail through it too |
pooler terminating connection due to administrator command | compute resize down underway (196 s measured, D04, 2026-08-04); on 2026-10-23 restarts and resizes up both answered it too, and resizes down ran 150 to 249 s on the longest path (one answered timeout expired), so it does not name the operation by itself | wait and retry on every path; REST reads and Realtime subscriptions fail through it too |
pooler (EADDRNOTALLOWED) address not in tenant allow_list | a network restriction was just applied and excludes this client (bites in 1 s, D02) | add the client’s egress address to the allow-list |
pooler XX000 (ECHECKOUTTIMEOUT) unable to check out connection from the pool after 60000ms in Transaction mode | transaction pool exhausted (the project is otherwise up): the connect succeeds in 43-52 ms and the first statement fails after about 60 s (60,007-60,009 ms, 6 of 6, Micro, PC01) | fall back to the session pooler on 5432 (49-70 ms from the failure to the fallback’s answer, PC02a) or add the direct or dedicated path (126-168 ms with the IPv4 add-on, PC02) |
pooler (ECHECKOUTTIMEOUT) unable to check out connection from the pool after 15000ms in Transaction mode (or in Session mode) | the pool had a slot, but the backend connection it handed over was not ready within a fixed 15 s: reproduced with Postgres logins delayed 20 s, in both modes, on Micro and Small, alongside 08006 ... authentication did not complete within 15000ms (PC06, 2026-10-23); with logins delayed 8 s every client was served after about 8 s | look at the database side (slow logins, a backend that cannot be reached); session mode on 5432 answered the same error, in Session mode, in PC06 (pooler-checkout RUNLOG) |
pooler (ECIRCUITBREAKER) too many authentication failures | the pooler’s circuit breaker: a wrong-password poll every 4 s opened it within about a minute, after which it refused the correct password too, before checking the password (restore-paths RP02 first attempt); also seen on failed JIT logins retried every 2 s for 60 s (JA01h) | poll a known-bad credential at most once a minute; read the breaker answer as no answer about the platform or the password |
The pool-exhaustion row was measured 2026-10-10 on one Micro project in
ap-southeast-1 (PC01, PC02). The effective transaction pool was 16
active backends while the API reported default_pool_size null, and a
statement that queued for less than 60 s was served (6,018 to 6,056 ms
behind 8 s holders, 4 of 4). A public status
mirror quotes after 15000ms for an incident on another cluster on
2026-08-24;1 this
project answered 60000 ms in 6 of 6 trials. On 2026-10-23 (PC01, PC05-PC07,
Micro and Small) the 60,000 ms pool-slot wait did not move with a node-postgres
statement_timeout, connectionTimeoutMillis or a pool size of 5, and
after 15000ms was reproduced by delaying Postgres logins past 15 s: the
figure names which wait expired, the pool slot (60 s) or the backend handover
(15 s). That public reports were the handover path is inference; no project
with a 15 s pool-slot wait was found. Read it off the error text. The session pooler took
one extra backend while the 16 transaction-mode backends were held (17
active pg_sleep backends, PC02h, n=1); a herd of fallbacks was not
tested, and each fallback client holds a backend for its whole
connection. The direct host did not resolve from an IPv4-only client
without the IPv4 add-on (ENOTFOUND, 6 of 6, PC02b), and the add-on took
80 s from the PATCH to an IPv4 address on that resolver in both runs, so
treat it as a setup step. On the breaker, the JIT
run listed one banned IPv4 address through POST /network-bans/retrieve
and DELETE /network-bans lifted it; its pooler rows after that point
were discarded (JA01h, run 1).
The restart row was re-checked on one Micro project with five restarts
in sequence (LO05, 2026-10-10, 0.5 s sampling; the D01 numbers stand as
measured). Window p50 and max were 54 and
60 s on Auth, 58 and 59 s on Storage, 54 and 151 s on the pooler and 50
and 56 s on the direct path, with the first failure 2 to 3 s after the
POST. Runs 1 and 2 ran about 30 s on every path except the pooler’s 151 s
in run 2, and runs 3 to 5 ran 50 to 60 s; one session cannot say whether
that is drift, load or chance. The project status read ACTIVE_HEALTHY
again at 40 to 71 s, and in run 2 it read healthy at 41 s while the
pooler window ran to 151 s, so the status call does not mark the end of an
outage. That run’s REST probe was an anon GET /rest/v1/, which the API
gateway answers with 401 before PostgREST or the database is reached, so its
0 failed runs of 5 said only that the gateway answered. Re-run on 2026-10-23
with a row read and a Postgres Changes join (LO05 artifact),
REST reads and Realtime joins failed in 5 of 5 restarts, p50 150 s and
146 s, max 181 s and 179 s, alongside Auth, Storage, the pooler and direct
(p50 145 to 151 s). Those p50s ran more than twice the 2026-10-10 p50s
(50 to 58 s), though runs 3 and 4 (43 to 58 s) matched them; not explained. This is why the probes above read a canary table.
The pooler error text varied inside these restarts:
{:error, :timeout}, terminating connection due to administrator command and (EAUTHQUERY) auth_query secret check timed out. Restart,
resize and restriction costs side by side are in what a platform
operation costs.
Detection also has a non-engineering half. Subscribe to status.supabase.com so a platform incident reaches you before your probes finish arguing about it, and know what support your plan includes. Pro gets access to the support team; Team and Enterprise carry support SLAs (Urgent: 24h 24/7x365 on Team, 1h 24/7x365 on Enterprise; business-hours limits start at High - the full severity table is on the SLA page). And on Team or Enterprise, give support at least 2 weeks’ notice before a launch or heavy load event via the support form, which is the documented way to ask support to watch your project during that window.
The status page can also gate your own changes, as an optional check on
the probe side (LO01, experiments/lifecycle-ops/lib/gate.ts; one fetch
of the JSON feed at 00:43 UTC on 2026-10-10 and a replay, so read it as a
study of the feed’s shape). Three properties of the feed set the rules.
status.supabase.com/api/v2/components.json carries each of the 18 region
names 10 or 11 times (189 components, no group or service field), so a
lookup by region name takes the worst status over every match.
incidents.json has no component or region field and
incidents/unresolved.json answers 404, so read the region scope from
the incident text, title first. impact is ignored: replayed over the
25-incident feed, the gate refused 11 incidents for an upgrade, 3 of them
published with impact none, and 10 for a restart, 2 of them with none.
Component status has no positive case in the lab yet: all 189 components
read operational on the day, so whether any of them flips during a
regional incident was not measured. Over the feed’s 1113 hours the replay
would have refused a restart in ap-southeast-1 for 38.5 h, on a text
heuristic. Failing closed when the page cannot be
read is a design choice and was not measured.
Part 2: Exposure reduction - the jwt_exp lever
Section titled “Part 2: Exposure reduction - the jwt_exp lever”A JWT-skew incident only rejects tokens minted during the skew window. Tokens minted before it keep working, so a longer access-token TTL shrinks the cohort washed through refresh during the window.
# Read current value (default 3600)curl -s -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ "https://api.supabase.com/v1/projects/$REF/config/auth" | jq '.jwt_exp'
# Raise to 12h (issuer-effective in ~6.5s, not instantly)curl -s -X PATCH -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ "https://api.supabase.com/v1/projects/$REF/config/auth" \ -d '{"jwt_exp": 43200}'Poll the readback after the PATCH, and again after the restore: the apply
is async, and a readback issued immediately after a restore PATCH returned
the pre-restore value (edge-resilience W17, 2026-08-17 battery; the module
now polls). Then verify the effect, not the status code - issue a token
and decode exp - iat. Use the admin path, because hosted
/auth/v1/signup hits email rate limits under scripted load:
curl -s -X POST -H "apikey: $SECRET_KEY" -H "Authorization: Bearer $SECRET_KEY" \ -H "Content-Type: application/json" \ "https://$REF.supabase.co/auth/v1/admin/users" \ -d '{"email":"probe@example.com","password":"probe-pass-1","email_confirm":true}'curl -s -X POST -H "apikey: $PUBLISHABLE_KEY" -H "Content-Type: application/json" \ "https://$REF.supabase.co/auth/v1/token?grant_type=password" \ -d '{"email":"probe@example.com","password":"probe-pass-1"}' | jq -r .access_tokenLonger-lived tokens are also longer-lived if leaked. 12h suits apps whose enterprise clients already hold day-long sessions.
Part 3: Read survival - the edge Worker cache
Section titled “Part 3: Read survival - the edge Worker cache”An edge Worker with a cache keeps warm reads answering through an origin outage: the drill measured 200 with a byte-identical body while the origin was hard-down. Origin-first (fetch the origin, serve cache only on failure) keeps reads fresh and still absorbs the outage; use it for a Data API serving live rows. Cache-first (a hit short-circuits before the origin) serves reads up to max-age old in normal operation, so use it only for mostly-static paths. The example below is origin-first. The lab’s drill worker is cache-first, because it was built to measure outage survival and freshness was not under test.
export default { async fetch(req: Request, env: { UPSTREAM: string }, ctx: ExecutionContext) { const url = new URL(req.url); if (req.method !== "GET") return fetch(new Request(env.UPSTREAM + url.pathname + url.search, req));
const cache = caches.default; const key = new Request(url.toString(), { headers: req.headers });
// PostgREST treats unknown query params as column filters and 400s. // Strip probe-only params (this example: anything starting with "_") // from the origin URL; keep the full URL only in the cache key. const originUrl = new URL(url); for (const k of [...originUrl.searchParams.keys()]) { if (k.startsWith("_")) originUrl.searchParams.delete(k); }
// Origin-first: fresh reads in normal operation, cache as the outage // fallback. (For a mostly-static path, check cache.match(key) first // and short-circuit on a hit - that is the cache-first posture.) let origin: Response; try { origin = await fetch(new Request(env.UPSTREAM + originUrl.pathname + originUrl.search, req)); } catch { const stale = await cache.match(key); return stale ?? new Response('{"error":"origin unreachable"}', { status: 503 }); } // CF Workers wraps TCP failures to unroutable origins as a 403 // RESPONSE, not a throw - include 403 here if an unroutable origin // is your failure mode (it is the drill's). if (origin.status >= 500) { const stale = await cache.match(key); if (stale) return stale; return origin; } if (origin.ok) { const toCache = origin.clone(); // The gateway sets Set-Cookie (__cf_bm) on EVERY response and the // Cache API refuses Set-Cookie responses - strip or nothing caches. toCache.headers.delete("set-cookie"); toCache.headers.set("cache-control", "public, max-age=86400"); ctx.waitUntil(cache.put(key, toCache)); } return origin; },};A third option is stale-while-revalidate (serve the hit, refresh in the background) when you want both freshness and latency. None of the three invalidates on write; if clients write through the same Worker, delete the cache key on mutations.
Deploy with wrangler (wrangler deploy); set UPSTREAM to
https://<ref>.supabase.co. Scope the cache to read-only paths you mean to
cache - do not cache auth or per-user responses unless the cache key
carries the caller’s identity. The example passes req.headers into the
key Request, which does not make the key per-user: the Cache API
documentation
lists the response headers put() respects and says nothing about request
headers as key material. Fold a hash of the Authorization header into the
key URL yourself, or keep the Worker on anonymous paths (the lab’s W04
drill cached only those). Not measured in the lab: no module cached a
per-user response.
The lab worker extends this into an active-passive failover proxy,
configured by env vars FAILOVER_PRIMARY / FAILOVER_STANDBY /
HOLD_MS. On a primary failure (5xx, the CF-wrapped 403, or any non-ok
in outage mode) it re-fetches from the standby and tags the response
x-drill-origin: standby. After a failure it holds on standby for
HOLD_MS even if the primary recovers, to damp flapping, and persists
the last-failure timestamp in the Cache API so the window survives
redeploys. The measured drill set two rules. Failover mode skips the
cache-first read, because a HIT carries no origin information and would
mask the failover. The hold window must exceed your redeploy-plus-settle
path: ~11s was observed, and the drill ships 60000 after 15000 measured
an expired window. Full code:
worker.ts;
the drill that drives it end to end:
w24-edge-failover-proxy.ts.
Part 4: Full HA - warm standby with session-portable cutover
Section titled “Part 4: Full HA - warm standby with session-portable cutover”Each step below was measured; the full routing-flip RTO end to end was not. Replication lag was 34ms-1057ms cross-region, sessions carry over via third-party auth, and the cutover can be rehearsed.
Run the standby at the compute size you will need at cutover, ahead of
time, because a resize during the incident adds its own window to your
RTO. Across four resizes (compute-disk D09, 2026-10-23, 250 ms sampling,
n=1 per op) REST reads, Auth and Realtime joins were down 144.528 to
254.886 s per resize: 231.465 s on Micro -> Small and 223.28 s on Small ->
Large. The status read ACTIVE_HEALTHY 136 to 245 s after each PATCH and
REST reads came back 18 to 35 s after that. In platform-downtime at 500 ms
sampling, a Micro -> Small step took 131 s on Auth and 207 s on the pooler
(D03, 2026-08-04). If you must resize twice, space the PATCHes by at least 2 minutes; the second inside
that window answered 429 We are still processing addon changes, please try again in 1-2 minutes (D09).
4.1 Replicate
Section titled “4.1 Replicate”On the primary (single statements - see gotchas):
create publication ha_pub for table public.<your_tables>;Before the subscription, check the slot budget on both sides.
show max_replication_slots returned 10 on Micro and 10 on Small
(compute-disk D01), and each subscription pins one slot on the publisher.
A dropped subscription can leave its slot on the publisher either way,
after a plain drop (W05) and after slot_name = none then drop (W14), so
list pg_replication_slots on the publisher and
pg_drop_replication_slot any orphan first.
On the standby, same table DDL first (DDL does not replicate), then the subscription against the primary’s direct host - never the pooler:
create subscription ha_sub connection 'host=db.<PRIMARY_REF>.supabase.co port=5432 dbname=postgres user=postgres password=<DB_PASSWORD> sslmode=require connect_timeout=15' publication ha_pub;Verify that seed rows appear on the standby (initial sync ~3.1-6.5s on a
small table; 22.7s/12.5s measured for 1M rows), then insert a canary and
time its appearance (34ms-1057ms cross-region; ~245-276ms right after a
large initial sync). Public and custom schemas replicate; auth.* and
storage.* never do, and on those platform-managed schemas the sync never
completes: with copy_data=true it hangs in pg_subscription_rel state d
(sync worker stuck at IPC/BgworkerStartup, its publisher sync slot
inactive), and with copy_data=false the WAL sender connects but
received_lsn stays NULL - zero changes stream, at any size (edge-resilience
W14, 2026-08-15, a manual drill). Instance size is not the variable:
max_worker_processes is 6 on both micro and small.
4.2 Make sessions portable
Section titled “4.2 Make sessions portable”Register the primary’s issuer on the standby once, at setup time (not at incident time):
curl -s -X POST -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ "https://api.supabase.com/v1/projects/$STANDBY_REF/config/auth/third-party-auth" \ -d "{\"oidc_issuer_url\": \"https://$PRIMARY_REF.supabase.co/auth/v1\"}"Primary-issued tokens then validate against the standby’s API (measured
200). The first time the standby’s PostgREST sees the issuer’s key it
answers PGRST301 for ~30s, so rehearse before you need it. This keeps
existing sessions alive on reads. Fresh logins on the standby hit the
standby’s own auth store, and auth.* replication between managed
projects is a measured dead end (zero changes stream at any tested size,
micro or small, with or without copy_data). Reads and existing sessions
survive; new logins need a user backfill or a forced re-login after
cutover. There are two backfill paths: POST /auth/v1/admin/users
accepts the source’s bcrypt password_hash and the user keeps their password
(measured, tenant-consolidation C03), or copy hashes via direct SQL into
auth.users (not measured on a standby); otherwise accept the re-login.
Per-project auth
config (SMTP, SITE_URL, redirect URLs, rate limits, jwt_exp) does
not follow a cutover either; re-apply it to the standby via the
Management API as part of the procedure below (measured: the diff is
readable verbatim from GET /v1/projects/{ref}/config/auth).
If your identity source is an external IdP via third-party auth or SAML SSO, both projects can trust the same issuer and fresh logins work on both sides. The backfill/re-login caveat above applies to Supabase-managed password users; accounts already federated skip it.
4.3 Cutover procedure
Section titled “4.3 Cutover procedure”- Confirm replication lag is near zero (canary insert appears < 2s; measured ~245-276ms immediately after a 1M-row initial sync).
- Freeze writes on the primary (fail closed; never dual-write).
- Resync sequences on the standby (sequences do not replicate - the
first insert without this step fails verbatim:
duplicate key value violates unique constraint "<table>_pkey", measured):
select setval(pg_get_serial_sequence('public.<t>','id'), coalesce((select max(id) from public.<t>), 1));- Flip routing (DNS, or your edge Worker’s routing table) to the standby.
- After stabilisation, drop the old subscription and the orphaned replication slot on the old primary:
-- standbydrop subscription if exists ha_sub;-- old primaryselect pg_drop_replication_slot('ha_sub');- Re-apply per-project auth config on the standby: GET
/v1/projects/{ref}/config/authon both projects, diff the entire payload, PATCH the standby. SMTP,SITE_URL, redirect URLs, rate limits andjwt_expare the fields known to matter, and the list is not closed:custom_oauth_max_providersdefaulted to 32767 on one project generation and 3 on another (W17 baseline diff, 2026-08-17). Poll the readback after the PATCH (the apply is async - Part 2). - Reverse the replication direction to fail back (not measured - no module runs the reverse direction).
A PITR or daily-backup restore of the primary unwinds this topology - subscriptions and replication slots must be dropped before a restore and recreated after (only the Realtime slot is exempted), so the standby’s subscription dies with a primary restore and must be rebuilt with a re-sync. Mid-incident, restore-to-a-new- project keeps the original (and the replication) up instead of taking the size-dependent restore downtime.
No Management API route for it was found (checked 2026-10-10 on one Pro
org, RP01). The published OpenAPI document (115 paths) has no
path matching clone, the create-project body has no clone, restore,
source or backup field, and the Dashboard’s clone route answers a
personal access token with 401 Unsupported access token. The
entitlements project_cloning and backup.restore_to_new_project read
true on that org, so the feature is entitled and its route is the
Dashboard’s; it was not run, and the database password and Storage after a
clone are unread. POST database/backups/restore with a backup id
answered 400 This endpoint is unavailable at the moment on a fresh Pro
project. The in-place route that ran is POST database/backups/restore-pitr
(PITR add-on on a Small project, RP02, 2026-10-10): the project returned
to ACTIVE_HEALTHY after 30 s with a 10 MB database (n=3, 10 s polling)
and after 81 s at 574 MB (n=1). The HTTP-tier REST probe (the anon
GET /rest/v1/ the gateway answers with 401), Auth and the Realtime
handshake showed no failed 1 s sample across the 9 clean restores (one
unexplained simultaneous timeout on all three in the 574 MB run aside),
Storage failed for 12 s (8 of 9 restores), and the pooler for 5 to 13 s on
10 MB and 41 s at 574 MB. A Data API read was not probed in those runs; in
later restores that probed one (restore-paths RUNLOG,
RP06 and RP07, 2026-10-22 and 2026-10-23) it answered 503 PGRST002 for 15
and 14 s on two small restores and for 192 to 544 s on seven restores of 3.2
to 10.5 GB databases (the test machine slept during these runs - checked, lab #234: range ends stand, Auth/Realtime windows void)).
After each of the 9, the current database password logged in and the
replaced one was refused, and the stored password verifier matched
neither earlier value, which fits the platform’s 2026-07-30 credential
resync fix2 (inference from fingerprints). Every target was minutes old; a target
hours or days old was not run. An in-place restore rewinds the primary,
so it unwinds the standby as described above. The full backup/PITR
picture is in the DR tiers reference.
When restoring into a new project (from the Dashboard), wait for
ACTIVE_HEALTHY and then retry the first write: 5 of 5 fresh
projects refused the first POST /auth/v1/admin/users with 500 and
accepted the next attempt one poll tick later. That figure is the
placement reference’s Verified row ‘Create -> healthy, and healthy is not
writable’ (2026-08-04, n=5, a bash run not in the lab repo). The lab’s
own provisioning note in supabase-lab AGENTS.md records 2 of 2 fresh
projects failing with 500 "Database error checking email" on
2026-08-03. Both are summarised in
what a platform operation costs.
Three creates through POST /v1/projects on 2026-10-10 (LO03, 10 s poll)
read ACTIVE_HEALTHY at the first poll twice and at 145 s once, and the
project that read healthy at the first poll (LO05a) answered REST, Auth,
Storage, the pooler and the direct path within 5 s. The lab recorded no
first write on those projects, so the two sets of observations are not
reconciled; keep the retry.
4.4 Rehearse
Section titled “4.4 Rehearse”Run the cutover monthly against a canary workspace. The measured numbers say the mechanics cost seconds; the cold-path and forgotten-step costs are what rehearsals find.
Part 5: Cold DR floor
Section titled “Part 5: Cold DR floor”On paid plans you already have daily backups (Pro 7 days, Team 14, Enterprise 30) and PITR as an add-on - the tiers, costs and restore caveats are in the DR tiers reference. The cron’d pg_dump below is the floor for free-tier projects, and for data too big to replicate or too cold to matter hourly:
# The dump runs over the pooler session host - session mode on 5432 is a# legitimate dump path (and the W06 measurement above went through it);# the direct host db.<ref>.supabase.co works too.# The aws-<N>- index is a per-project pooler-cluster number, not always 0:# read the host from GET /v1/projects/<ref>/config/database/pooler rather# than assuming it.PGPASSWORD=$DB_PASSWORD pg_dump -h aws-<N>-<region>.pooler.supabase.com -p 5432 \ -U postgres.$REF -d postgres -Fc -f backup-$(date +%F).dumpMeasured floor: 12.4s dump / 6.4s restore for 10k rows. Schedule it; ship the file to object storage outside the incident blast radius.
What to do about it
Section titled “What to do about it”Each practice names the module or RUNLOG line it rests on; module ids resolve in the edge-resilience RUNLOG, the platform-downtime RUNLOG, the compute-disk RUNLOG, and, for the 2026-10-10 additions, the RUNLOGs of pooler-checkout, lifecycle-ops, restore-paths, client-retries and jit-db-access. A line marked “design choice” is not a result.
Detection (Part 1)
- Probe Auth through a real operation (admin-create plus password grant)
rather than
/auth/v1/healthalone. This is a design choice; both lab runs sampled/auth/v1/health, down 75 s on a restart (platform-downtime D01) and 152.375 to 254.869 s across four resizes (compute-disk D09, 2026-10-23), and no module measured an authenticated Auth path. - Probe Storage against a real object, and on a fresh project expect
TenantNotFoundthen429 SlowDownfor the first minutes (W21). - Probe the pooler with
select 1over 6543 under a short connect timeout, and read the error string against the pooler rows in the Part 1 table (platform-downtime pooler-mode table): on 2026-08-04 each operation had its own string; on 2026-10-23 restarts and resizes up both answeredterminating connection due to administrator command, so onlyEADDRNOTALLOWEDnames its operation by itself. With the lab’s 5 s probe timeout a timing-out pooler was resampled about every 5.5 s, which is the resolution its 158 s figure carries. - Declare recovery after sustained success and void rows that were failing before the operation began (platform-downtime “Reproducing”).
- Run a statement in the pooler probe, and read
ECHECKOUTTIMEOUTwithafter 60000msas pool exhaustion: the connect took 43 to 52 ms and the first statement failed at 60,007 to 60,009 ms (6 of 6, PC01, Micro, 2026-10-10). Readafter 15000msas a backend connection that was not ready: delayed logins reproduced it at 15,051 to 15,081 ms in transaction mode (PC06, 2026-10-23). The session pooler on 5432 answered 49 to 70 ms after the failure (PC02a); the direct host needs the IPv4 add-on from an IPv4-only client (PC02b), which took 80 s to resolve (PC02). - Retry a statement that failed on a reset pooler connection after 300 ms,
not 0 ms, and attach an
'error'listener to a node-postgresPool: 25 of 25 and 30 of 30 rescued at 300 ms against 0 of 30 at 0 ms after a 150 ms flap, and 9 of 12 runs exited without the listener (PC03). - Poll a known-bad credential at most once a minute and read
ECIRCUITBREAKERas no answer: a poll every 4 s opened it within about a minute and it then refused the correct password (restore-paths RP02, the aborted first attempt, no artifact), and a failed JIT login retried every 2 s for 60 s did the same (JA01h). - Gate a change on the status page by the worst status over every
component named for the region, with the scope read from incident text
and
impactignored (LO01, one fetch). Failing closed when the page cannot be read is a design choice, and no component status has been seen other thanoperational. - Budget a restart at about a minute (LO05, 2026-10-10, p50 50 to 58 s per
path) to about 2.5 minutes (LO05 re-run, 2026-10-23, p50 145 to 151 s, max
182 s) on every path, REST reads and Realtime included, and do not read
ACTIVE_HEALTHYas the end: 40 to 71 s on the status call on 2026-10-10, 41 s in the run whose pooler window ran to 151 s, and 41 to 172 s on 2026-10-23, each time before the row read recovered (LO05, n=5 per date, one Micro project each). - Expect supabase-js to retry only GET and HEAD on 503 and 520, and set a
timeout yourself: a default client had none, and a request never
answered was still pending at 330 s on Bun and on Node (CR04, one trial
each, 2.112.3). Under Bun 1.3.14 an inline
AbortSignal.timeoutdid not end a call sleeping between retries; anAbortControllerwithsetTimeoutdid (CR01d5a-b).
Exposure reduction (Part 2)
- Poll the readback after every
config/authPATCH, including the restore; an immediate readback returned the pre-restore value (W17, 2026-08-17). The 12h value is a design choice; the ~6.5s lever is the measurement (W03).
Read survival (Part 3)
- Put the caller’s identity into the cache key yourself before caching any per-user response; the lab cached anonymous reads only (W04), so this is a design choice.
- The origin-first posture and
max-age=86400are design choices; the lab drilled cache-first (W04).
Warm standby (Part 4)
- Run the standby at the cutover compute size ahead of time: resizes
took REST reads, Auth and Realtime joins down 144.528 to 254.886 s each,
223.28 s on Small -> Large (compute-disk D09, 2026-10-23, n=1 per op),
and 131 s Auth / 207 s pooler on Micro -> Small in platform-downtime
D03 (2026-08-04). Space adjacent resize PATCHes by at least
2 minutes or the second answers
429 still processing addon changes(D09). - Check
max_replication_slots(10 on Micro and Small, compute-disk D01) and drop orphan slots beforeCREATE SUBSCRIPTION(W05, W14). - Never drop and recreate a table under an active Realtime subscription (events for that name stop; new OID, stale channel metadata), and retry joins with backoff - a join after 11 other modules had the socket closed before the event (W12).
- Diff the entire
config/authpayload at cutover, not five named fields;custom_oauth_max_providersread 32767 on one project generation and 3 on another (W17). - When restoring into a new project, wait for
ACTIVE_HEALTHYand retry the first write: 5 of 5 fresh projects refused it with 500 (placement reference, Verified row ‘Create -> healthy, and healthy is not writable’, 2026-08-04, n=5, bash run not in the lab repo), and 2 of 2 with500 "Database error checking email"in the supabase-lab provisioning note (2026-08-03). On 2026-10-10 three creates readACTIVE_HEALTHYat the first poll twice and at 145 s once (LO03), and a first-poll-healthy project answered all five paths within 5 s (LO05a); no first write was recorded, so keep the retry. - If the mid-incident restore has to be scripted, the Management API route
is an in-place
restore-pitr, which takes the restore downtime and unwinds the standby; restore to a new project was not found in the API and was not run from the Dashboard (RP01: no clone path in the OpenAPI document, the Dashboard route answers a PAT 401).restore-pitrreturned toACTIVE_HEALTHYin 30 s on Small with 10 MB (n=3) and 81 s at 574 MB (n=1), with Storage failing 12 s (RP02, 2026-10-10). The current database password still logged in after 9 of 9 clean restores. - Not measured: failing back by reversing replication (4.3 step 7); a primary-side network restriction against the standby’s walreceiver; the full routing-flip RTO end to end (per-step timings only: W05, W16, W17); the monthly rehearsal cadence (design choice).
Verification
Section titled “Verification”- Probes: kill nothing, but confirm each probe reports the right code when you hand it a bad token (PGRST301) and a future-iat token (PGRST303, mintable with the lab harness).
- TTL lever: issue a token after the PATCH, decode
exp - iat= new value. - Edge cache: prime a URL, block the origin (point
UPSTREAMat a dead host), confirm 200 from cache for the warm URL and failure for a cold one. - Standby: run 4.3 end-to-end on a canary workspace and time it.
- Failover: point the Worker’s primary at an unroutable address, confirm the response flips to the standby origin tag, restore, and confirm the holdover window keeps serving standby until it expires (the lab’s W24 drill automates exactly this, four redeploys included).
Gotchas and lessons learned
Section titled “Gotchas and lessons learned”CREATE SUBSCRIPTIONmust be a single-statement query. The Management query endpoint wraps multi-statement strings in one transaction, and Postgres rejectsCREATE SUBSCRIPTIONinside one. This presented as an unexplained SQL 400 until the verbatim body was read.- Config APIs are not request-path truth. TPA shows
resolvedwhile PostgREST still rejects the key (~30s cold);jwt_expreads back before the issuer honours it (~6.5s), and a readback taken immediately after a restore PATCH still showed the old value (W17, 2026-08-17); ajwt_secretPATCH returns 200 and changes nothing. Verify at the request path, always. - The gateway sets
Set-Cookieon everything. Strip it beforecache.putor the Cache API refuses silently. - Cloudflare Workers wraps TCP failures to unroutable IPs as a 403
response, not a JS exception - stale-fallback
catchblocks never see it. Branch on status too. For failover logic this means the trip condition must include 403, or an unroutable primary never fails over (measured 2026-08-16). - Cache-first and failover do not compose. A cache HIT carries no
origin information, so a failover mode behind a cache-first read gets
masked by warm entries. If the worker fails over between origins, skip
the cache while failing over and tag every response with the origin
that served it (
x-drill-originin the lab worker). A holdover window (hold on standby for N ms after a failure) damps flapping; persist the last-failure timestamp in the Cache API if it must survive redeploys. - PostgREST 400s on unknown query params (treated as column filters).
A
?_bust=cache-buster forwarded to the origin 400s every request - strip probe-only params from the origin URL, keep them only in the cache key (measured 2026-08-16; the Part 3 example does this). - Standby REST reads need the standby’s own publishable key, not the primary’s. They look interchangeable and are not.
- Wedged-subscription recovery is ordered:
alter subscription <s> disable->alter subscription <s> set (slot_name = none)->drop subscription <s>->pg_drop_replication_slot('<s>')on the publisher. The publisher slot survived the drop either way - after a plain drop (W05) and afterslot_name = nonethen drop (W14) - so the finalpg_drop_replication_slotstep is the one that frees the WAL; subscriptions can also vanish and reappear under platform management (measured during the W09/W14 drills). - Break-glass secrets stay live until rotated. Reading
jwt_secretthrough the Management API is auditable, but the secret does not expire on its own - rotate it after any break-glass use. - Hosted signup is not a scriptable probe surface - the default email
sender rate limits (
over_email_send_rate_limit). Admin-create plus password grant is the probe-safe path.
File reference
Section titled “File reference”| File | Role |
|---|---|
experiments/edge-resilience/worker/worker.ts | Reference Worker (cache + outage drill + failover mode) |
experiments/edge-resilience/tests/w05-standby-replication.ts | The standby drill, runnable |
experiments/edge-resilience/tests/w24-edge-failover-proxy.ts | The failover + flap-damping drill, runnable |
experiments/edge-resilience/FAILURE-MATRIX.md | The full failure-point inventory |
experiments/edge-resilience/RUNLOG.md | Every measured number, per module |
RELIABILITY.md (repo root) | The incident-class reference this guide implements |
Modules
Section titled “Modules”References
Section titled “References”-
Supabase status mirror, “Error spikes in one of the eu-east-1 Supavisor clusters,” 2026-08-24, Pulsetic. https://pulsetic.com/status/supabase/incidents/6635/ ↩
-
Supabase, “Fixed stale database credentials left behind after a restore,” Supabase Changelog, 2026-07-30. https://supabase.com/changelog/restore-credential-resync ↩
Linked from
- Monitor Supabase with Grafana: dashboards and alerting
- pgjdbc and HikariCP on Supabase connection paths
- Query Supabase project logs through the Management API after logs.all
- Running a platform on the Supabase Management API
- Supabase Auth providers: custom OIDC, passkeys, SAML and a sign-in canary
- Supabase disaster recovery tiers: daily backups, PITR, warm standby - RPO, RTO, and cost
- Supabase incidents: what a client can do - a resilience reference