Skip to content

SupabaseOperations and accessreference

Supabase incidents: what a client can do - a resilience reference

The platform fails in a small number of recurring ways. Each class below has a mechanism, a client-visible signature, and a workaround tier - and each claim is marked as measured in the lab (with the number) or documented-only. The workarounds are implemented in the resilience runbook; the RPO/RTO/monthly-cost axis for the durability tiers is in the DR tiers reference.

Measured numbers come from the supabase-lab pvlab harness - experiments/edge-resilience (2026-08-15/16, W21/W25/W26 2026-08-17) and experiments/platform-downtime (2026-08-04, n=1 per operation, one Micro project in ap-southeast-1); the cutover drill ran cross-region ap-southeast-2 -> ap-southeast-1.

Classes 2, 3, 4, 7 and 8 also carry measurements dated 2026-10-10 from six further experiments, each with a RUNLOG (the published out/ artifacts of three of them are linked in the Modules table): client-retries (supabase-js 2.112.3 and 13 other versions on Bun 1.3.14, faults injected by a local proxy, n=1 per case unless stated), pooler-checkout (Micro, ap-southeast-1 and us-east-1, Pro org), lifecycle-ops (one Micro project, ap-southeast-1, n=5 restarts), mgmt-api-faults (supabase CLI 2.120.0, OpenTofu provider, the lab’s own client, every fault injected, no real 5xx seen), hostname-path (one Pro project, ap-southeast-1, one APAC vantage) and edge-runtime-auth (the functions.invoke retry cell only). The Modules table maps each id to its test.

TL;DR: Durability has its own tier ladder, priced out in the DR tiers reference: daily backups (included, up to ~24h RPO), PITR (~2-minute worst-case RPO, ~$100-$400/month), warm standby (34ms-1s). Reads are cheap to protect (edge cache, measured: byte-identical through a forced origin outage). Writes and auth need a warm standby, which works managed-to-managed with logical replication and sessions that survive cutover via third-party auth (TPA, Supabase’s mechanism for trusting an external issuer’s tokens) registration (measured: 34ms-1s replication lag, TPA resolution ~60-122ms). Platform-managed schemas (auth.*, storage.*) do not replicate at any tested size, so standby auth posture is token portability plus forced re-login, not user replication. The JWT clock-skew incident class has no runtime workaround - only exposure reduction (a token TTL lever that takes effect in ~6.5s) and detection (the PGRST code, not the HTTP status). The built-in supabase-js retries cover GET, HEAD and rpc({ get: true }) on 503, 520 and connection resets only, with no default timeout (Class 8); a full transaction pool fails the first statement after 60 s while the connect succeeds (Class 3).

You need to survive…TierCostEvidence
Origin 5xx on read pathsEdge cache WorkerOne Worker, no infra changeMeasured, W04
Transient 503 or 520 on a GETBuilt-in supabase-js retries (2.102.0 and later)On by default; a deadline is yours to addMeasured, CR01-CR04
Transaction pool full (ECHECKOUTTIMEOUT)Session pooler on 5432 as fallbackClient-side switchMeasured, PC01-PC02
JWT claim rejections (PGRST303)TTL raise + code-aware alertingConfig changeMeasured, W01/W03
Auth outage with active sessionsLonger TTL (sessions keep validating)Config changeMeasured, W03
The whole PostgREST/Kong API layer (PGRST303, routing rollouts)Own auth + claims over the wireRewrites the client; loses SDK surfaceMeasured, rls-wire-claims C01-C03 (reference)
Full regional/project lossWarm standby + cutoverSecond project + rehearsalMeasured, W05
Anything elseCold DR (pg_dump)Cron + object storageMeasured, W06
Data loss after the factPITR (point-in-time restore)~$100+/mo + Small computeDocumented (tiers)
Bad DELETE / bad migrationPITR - the standby replicates the mistake in 34ms~$100+/moDocumented
Org-wide 402 restrictionFix billing (card / cap / usage)-Documented
Fleet-wide incidentContract (SLA)Enterprise planDocumented

Most readers need six to nine of the thirteen classes, depending on their shape.

Your shapeReadsWrites / authDurabilityClasses that apply
Free, one projectedge cache (7)accept the outagescheduled db dump - Free has no backups1, 2, 4, 5, 7, 12
Pro, one project, productionedge cache (7)break-glass mint (W07) + TPA prepdaily backups (7d) or PITR1-8, 12
Team / Enterprise, productionedge cache + failover (7)warm standby (9)PITR + standby (tiers)all
Enterprise, federated IdPsamestandby; fresh logins work via the shared issuerPITR + standbyall; the auth.* caveat drops out

Classes 3, 8, 10 and 11 are USAGE-gated, not plan-gated: they apply on any plan the moment you use the pooler, supabase-js, Edge Functions, or a query long enough to hit a timeout.

PostgREST validates iat/exp on every request with a 30-second clock-skew tolerance.1 If the issuer’s clock runs ahead of the validator’s, freshly minted tokens carry a future iat and are rejected - while tokens minted before the skew began keep working. Auth (/auth/v1/user) can return 200 while the Data API rejects the same token; the two validate on different hosts.2

The client sees 401 {"code":"PGRST303","message":"JWT issued at future"}, and the failure reaches every active session as its access token expires and refreshes - with the default 1-hour TTL, the whole user base goes through the broken path within one TTL.

Measured in edge-resilience W01 (2026-08-15), the documented tolerance is exact: +30s accepted, +31s rejected (401 PGRST303). Expired tokens reject with the same code; an unknown signing key rejects PGRST301. Two concurrent refreshes of the same token both return 200 (W08) - simultaneous reuse does not hard-fail.

The Supabase status page logged “401 errors due to JWT rejections” on 2026-08-14 (“newly refreshed JWTs being rejected by the API”), with an earlier “elevated JWT authorization errors” incident the same week and a matching community report three weeks prior.32 Three occurrences in a month.

  • Raise jwt_exp via the Management API to reduce exposure. Readback is immediate for the new value; the issuer honours it ~6.5s later (measured, W03). The apply is async, so poll the readback after any config/auth PATCH including the restore - on 2026-08-17 an immediate readback returned the pre-restore value (W17). A 12h TTL means a 20-minute skew window touches a small fraction of sessions instead of all of them; the 12h figure is a choice, not a measurement.
  • Alert on the PGRST303 rate. The HTTP status (401) is ambiguous; the code identifies the class.
  • Do not retry mid-skew; every refreshed token is equally future-iat. The default client makes exactly 1 attempt on a claim rejection (measured, W02, supabase-js 2.112.3).
  • No runtime fix exists; the repair is platform-side (PostgREST update plus NTP hardening). The only structural alternative is leaving the validating layer entirely: claims over the wire keeps per-user RLS with your own verifier as the only clock (measured, rls-wire-claims C01/C02) - a rewrite, not a mitigation.

On restart (platform-downtime D01/D03/D04, 2026-08-04, n=1 per operation, one Micro project in ap-southeast-1, 500ms sampling), the REST and Realtime probes had zero failed samples, Auth 75 s (HTTP 521), Storage 78 s (HTTP 500), pooler 158 s (Failed to connect to database: {:error, :timeout}). The lab’s summary file (RELIABILITY.md) carried REST ~10s, Storage ~26s, Auth ~75s as N=3 p50 dated 2026-08-06..08; the RUNLOG is the record, and no artifact exists for those figures. A resize costs more than a restart, by a different factor per path: Auth 131 s against 75 s (1.75x), pooler 207 s against 158 s (1.31x).

Those REST and Realtime zeros were withdrawn on 2026-10-23. The REST probe was an anon GET /rest/v1/, which the API gateway answers with 401 before PostgREST or the database is reached, and the Realtime probe a WebSocket handshake, so both said only that the HTTP tier answered. Re-measured with a row read and a Postgres Changes join on throwaway Micro and Medium projects (platform-downtime RUNLOG, section 2026-10-23), REST reads and Realtime subscriptions failed for about as long as Auth and Storage: restart 247 and 194 s, Micro to Small 171 and 218 s, Medium to Large 113 s for the read (a second run is void: the test machine slept inside it, lab #211). Only a network restriction left them alone. The per-path matrix is in what a platform operation costs.

Re-check on 2026-10-10 (lifecycle-ops LO05, one Micro project in ap-southeast-1, five sequential restarts, each after a readiness check, same 0.5 s sampling; a window runs from the first failed sample to the first sample of a 5 s sustained recovery). The D01 row is the n=1 figure above, kept as it was; the two are different days and different projects, so the table is not a controlled comparison.

PathD01, 2026-08-04, n=1LO05, 2026-10-10, n=5: window p50 / maxLO05 per run (s, runs 1 to 5)
REST (HTTP tier)0 failed samples0 of 5 runs failed-
Auth75 s54 s / 60 s32, 31, 54, 59, 60
Storage78 s58 s / 59 s28, 29, 58, 59, 59
Pooler (6543)158 s54 s / 151 s31, 151, 52, 54, 59
Direct (5432)not sampled50 s / 56 s28, 28, 50, 55, 56

The first failed sample came 2 to 3 s after the POST /restart on every failing path. The control-plane status read ACTIVE_HEALTHY again at 40, 41, 61, 61 and 71 s (10 s poll), so it ended before the pooler recovered in run 2 (41 s against a 151 s pooler window). Windows were about 30 s in runs 1 and 2 (the pooler’s 151 s in run 2 excepted) and 50 to 60 s in runs 3 to 5; five runs on one project cannot say whether that is drift across back-to-back restarts, platform load or chance. Realtime was not sampled in LO05, and D01’s 158 s sits near LO05’s single 151 s pooler run, not near its other four. LO05 re-ran on 2026-10-23 with a row read and a Postgres Changes join (artifact): REST reads and Realtime joins failed in 5 of 5 restarts, p50 150 s and 146 s, max 181 s and 179 s, alongside Auth (p50 149 s), Storage (151 s), the pooler (147 s) and direct (145 s), while the HTTP-tier probes failed in 0 of 5. That day’s p50 windows (145 to 151 s) ran more than twice the 2026-10-10 p50s (50 to 58 s), though runs 3 and 4 (43 to 58 s) matched them; not explained. POST /restart answered HTTP 200 each time; a second POST /restart sent while the status read RESTARTING also answered 200 with an empty body (LO05c, whether it queued was not sampled). A restart sent about 70 s after the creating POST, to a project that already read ACTIVE_HEALTHY, was refused with HTTP 400 Project restarts are only allowed ten minutes after the creation process has completed. followed by text the artifact cut off; a restart 300 s after the first ACTIVE_HEALTHY read on another project was accepted (LO05f), so the boundary is unresolved. Probes on the macOS test machine used dns.resolve6 for the AAAA-only direct host, because dns.lookup (getaddrinfo) returned ENOTFOUND for it while a TCP connect to the literal address worked.

For probe design, anonymous network probes see nothing (network accept is not readiness), and the Auth probe must be the signUp REST surface - RLS blocks the password grant on a fresh project, so a password-grant sensor reports an outage that is really its own permission model. pg_cron jobs resume across a project restart and keep firing on schedule - a 1/min heartbeat job had 2 rows before the restart and 5 after (3 new rows across the window), with no catch-up doubling (measured, edge-resilience W23, 2026-08-16). The other side of no catch-up: an instant that falls inside the restart is skipped, and a run in flight is recorded failed and not re-run (cron-behaviour CB04/CB05, 2026-10-28; see pg_cron behaviour).

Measured in pooler-semantics and http-tier-lockdown, a session-mode startup auth failure is a clean 58P01 at connect. In transaction mode, connect succeeds, the first statement fails 08P01/0A000, and prepared statements are unsupported. The same HTTP 400 can be a bad request or a capacity ceiling - read the SQLSTATE. Non-pooler 5432 can accept and then silently blackhole; always set connect and statement timeouts.

Measured 2026-10-10 in pooler-checkout (PC01, PC02; one Micro project per run, ap-southeast-1, Pro org, Supavisor transaction pooler on 6543, one IPv4-only vantage in Singapore). The Management API reports default_pool_size null for the project, so the 16 below is a measured concurrency; no configured value was read back.4

ObservationValueModule
Active pg_sleep backends 6 s into 40 concurrent pg_sleep(25) clients16 in both runsPC01b
Client 17 (16 backends held by pg_sleep(75)): connect()succeeds in 43 to 52 msPC01c
Same client, first select 1fails after 60,007 to 60,009 ms (6 of 6 trials)PC01c
Error textXX000 (ECHECKOUTTIMEOUT) unable to check out connection from the pool after 60000ms in Transaction modePC01c
Holders sleeping 8 s, client 17 sends select 1 2 s inserved after 6,018 to 6,056 ms (4 of 4), no errorPC01d
select 1 after the holders ended7 to 8 msPC01c

A connect probe or a ping on connect does not see exhaustion; the client is accepted and the error arrives on the first statement. A statement that queues for less than 60 s is served. A public status-page mirror quotes the same error with after 15000ms for an incident on 2026-08-24 in one cluster;5 this project answered 60,000 ms in 6 of 6 trials. A 2026-10-23 re-run on a Micro and a Small project (PC01, PC05-PC07) separated the two figures: the after 60000ms pool-slot wait held at 60,006 to 60,013 ms whatever the client set (node-postgres statement_timeout, connectionTimeoutMillis, a pool size of 5), and after 15000ms came back in transaction and session mode when Postgres logins were delayed 20 s, at 15,046 to 15,180 ms, with the pool not full. The Supavisor source at commit d875a44f8316 (read 2026-10-22; the release serving the projects was not read) puts a fixed 15,000 ms on handing a slot’s backend connection to the client; that the public reports were this path, and not a tenant configured with a 15 s pool wait, is inference6. Role postgres cancelled a pg_sleep(130) with canceling statement due to statement timeout at 120 s in an exploratory run (not an artifact row), so holders must sleep under 120 s.

Fallback, with the 16 transaction-mode backends held (switch time is from the primary’s failure to the fallback’s answer; three trials per row in each of two runs):

FallbackSwitch timeModule
Session pooler, 5432, after a 3,000 ms client deadline49 to 70 ms (total 3,092 to 3,131 ms)PC02a
Session pooler, 5432, after the server’s own 60 s error56 to 117 ms (total 60,115 to 60,359 ms)PC02c
Direct 5432, no IPv4 add-on0 of 3 answered in each run; ENOTFOUND in 6 of 6 trialsPC02b
Dedicated pooler (PgBouncer) 6543, IPv4 add-on on126 to 167 msPC02e
Direct 5432, IPv4 add-on on135 to 168 msPC02f

The OS resolver on this vantage returned 0 IPv4 and 0 IPv6 addresses for the direct host without the add-on, while a public resolver (1.1.1.1) returned 1 AAAA record (PC04d), so the ENOTFOUND is this vantage’s view of an IPv6-only host. With the add-on enabled through PATCH billing/addons, this resolver returned an IPv4 address after 80 s in both runs, the dedicated pooler answered at 81 s and 80 s and direct at 81 s and 81 s (PC02d). One session-mode client running pg_sleep(15) beside the 16 held transaction backends took a 17th backend and completed (PC02h, n=1, pool setting unchanged), so one session-mode client fit as a 17th backend; it was not compared with a documented combined limit, and a herd of fallbacks was not run. Session mode behaves like a direct connection for prepared statements and SET, and each fallback client holds a backend for its whole connection.

PC03 put toxiproxy between the client and the transaction pooler and ran 5 workers issuing select 1 every 200 ms for 22 s in a pool of 5, with a fault 2 s long (reset, cut) or 150 ms long (flap), 6 runs per cell. After the fault cleared, 0 first attempts failed in the 111 runs that did not crash, for node-postgres, postgres.js and Prisma: all three evicted dead connections and recovered without a restart.

Driver, faultRescued by retry-once at 0 msRescued by retry-once at 300 ms
node-postgres, flap (150 ms)0 of 3025 of 25
Prisma, flap (150 ms)0 of 3030 of 30
node-postgres, reset2 of 2540 of 120
Prisma, reset2 of 26330 of 120
postgres.js, reset30 of 3030 of 30

A retry for a stale-connection failure has to wait longer than the outage it follows: at 0 ms it landed inside the 150 ms the proxy was down (an inference from the timing, not a separate probe). During a 2 s reset or cut, node-postgres and Prisma failed 40 to 50 first attempts per run at 0 ms and 20 per run at 300 ms; postgres.js showed no failed first attempt in 9 of 12 cut runs, with the slowest statement at 1,902 to 2,103 ms, about the length of the cut. node-postgres Pool without an 'error' listener exited non-zero in 9 of 12 runs at the cut (Unhandled 'error' event ... Connection terminated unexpectedly); with the listener 0 of 36 runs crashed. Not measured: a real Supavisor node replacement, prepared statements (postgres.js ran with prepare: false), idle pools, TLS verification, other driver releases.

On 2026-10-10 a fresh Pro project in each of two regions (ap-southeast-1 in runs 1, 2 and 5, us-east-1 in run 7) reported an aws-0 pooler host, and the same tenant on the aws-1 host answered XX000 (ENOTFOUND) tenant/user postgres.<ref> not found (535 to 648 ms, and 3,636 and 3,684 ms in us-east-1) while the aws-0 host answered (PC04). The hypothesis that aws-0 hosts are the old ones is not supported by these four projects; older projects and other regions were not measured. Read db_host from GET /v1/projects/{ref}/config/database/pooler rather than writing the host into code. The lab’s lint flagged 2 of 2 aws-0 literals in its fixtures and ignored dedicated and direct hosts.

Class 4: Capacity and control-plane events

Section titled “Class 4: Capacity and control-plane events”

The 2026 public status history documents recurring multi-region project creation/resize/restart failures: April APAC (~1.3h), April us-east (~2h), June multi-region (“existing projects are not affected unless restarted or resized”), August us-east-2 project access.3 Existing projects usually keep serving.

Pre-provision a project pool if your product provisions per-tenant, and never couple app availability to control-plane availability. There is no runtime fallback.

Your own billing state belongs in this class. Under the Fair Use Policy, continually exceeding the Free quota, exceeding the Pro quota with the spend cap on, or overdue invoices restrict the whole org: projects paused, databases read-only, launches blocked, and every API request answered 402 with a restriction description in the body.7 Detection is the 402 code in the probe table; the spend cap is not a request-path breaker (W21: quota+5 renders all 200); the fix is a card, a cap toggle, or reduced usage. The full billing/cost picture is in the DR tiers reference.

Measured in lifecycle-ops LO01 (2026-10-10, one fetch at 00:43 UTC, an earlier fetch at 00:33 UTC saw the same shapes, no credential). The status page’s JSON under /api/v2/ can gate a create, restart, resize or upgrade, with these shapes:3

WhatValue
Components in components.json189
Region names present18, each occurring 10 or 11 times; no group, service or parent field
Non-region components5 (Auth, Billing & Payments, Dashboard, Logging & Observability, Realtime)
Component statusoperational on all 189
Incidents in incidents.json25, oldest created 2026-08-24, 0 open; no component or region field
incidents/unresolved.jsonHTTP 404
summary.jsonone scheduled maintenance, created 2026-06-29, still scheduled on 2026-10-10

A lookup of the component named for a region therefore returns one of 10 or 11 components with nothing saying which service it is, so a gate takes the worst status over every match. All 189 read operational, so component status has no positive case in the lab yet. Incident scope is in text only: the gate (12 unit tests, fixtures) reads the title first and treats a region found only in an update body as a warning unless the text also has lifecycle wording, because one incident whose title named no region listed a dozen regions in a PostgREST rollout message and the first version of the gate counted it as 94.6 hours of refused restarts for ap-southeast-1. impact is ignored: replayed for upgrade the gate would have refused 11 of the 25 incidents, 3 of them published with impact none (including “Upgrade Issues”, whose update text says the ability to upgrade projects was suspended), and for restart it refuses 10, 2 of them with impact none. The gate does not read maintenance windows, since the one entry is 100 days old and still scheduled.

Replaying the 25-incident feed (1113 hours) through a restart gate, from incident text only because component status is not historical, would have refused ap-southeast-1 for 38.5 hours, eu-west-1 for 41.8 and us-east-1 for 205.1 (one incident, “Intermittent latency in Eastern US”, stayed open about 7 days); 38.5 hours was the minimum over the 18 regions. That classification is a heuristic and not ground truth. Not measured: component behaviour during an incident, how late the feed runs against an incident’s real start, and maintenance-window semantics.

Create, region fallback and abandoned requests

Section titled “Create, region fallback and abandoned requests”

Measured in lifecycle-ops LO03 (three creates) and LO04 (one), ap-southeast-1. Requests that must fail did so fast and created nothing: unknown region 400 (194 ms), unknown org slug 404 (80 ms), missing name 400 (93 ms), unknown instance size 400 (55 ms). A fallback wrapper that fell through one 400 landed on the second region; ACTIVE_HEALTHY came back at the first poll twice (2 s) and at 145 s once (10 s poll, n=3, so both happen and the count says nothing on how often). The project that read healthy at the first poll served REST, Auth, Storage and the pooler, and direct 5 s later (LO05a). A provisioning or capacity failure on the platform side cannot be induced from a client and was not run.

LO04 (n=1): a POST /v1/projects whose caller gave up after 1 s still created the project, and an identical re-send 1 s later answered HTTP 400 Project with name "<name>" already exists in your organization. after 0.17 s; 60 s later the organisation held exactly 1 project with that name, ACTIVE_HEALTHY. A client that retries after a lost response gets a 400 it has to read as “probably created, look it up by name”, and keeps the name stable across retries of one logical create; a wrapper that adds a random suffix per attempt would not meet that check (reasoned from the 400, not run). Not measured: case sensitivity, a name freed by DELETE, two concurrent identical requests, an idempotency key.

Management API clients under injected faults

Section titled “Management API clients under injected faults”

Measured in mgmt-api-faults (MF01-MF05, 2026-10-10, Pro org, ap-southeast-1): a fault-injecting proxy in a local container sat between each tool and api.supabase.com, with the CLI pointed at it through a profile file whose api_url is the proxy, the OpenTofu provider through endpoint, and the lab’s client through its base URL. Every fault is injected and no real 5xx was seen, so how a tool reacts to a real 5xx body is not measured. Attempt counts are proxy log rows; n=1 per cell unless stated.

ClientGETPOSTPUT, PATCH, DELETETimeout
Lab harness mgmt() (MF01)1 attempt on 500, 502, 503, 504; the status is returnednot runnot runthrows TimeoutError at its 30 s default (30001 ms)
supabase CLI 2.120.0 (MF03)6 attempts per call with no pause on 500 (also 502, 503, 504); 429 is 1 attempt1 attempt on 500, 502, 503, 504 and 429PUT and DELETE 6 attempts on 500; PATCH not reached by any command60 s per attempt; a GET held 100 s gave attempts at 0, 60, 120, 180, 240 and 300 s and exit 1 after 360220 ms
OpenTofu 1.13.1 with supabase/supabase 1.11.0 (MF04)4 attempts over about 6 s on a persistent 503; 2 on one 500 or one 4291 attempt on 5001 attempt on 500 (PATCH, DELETE)waited 130 s for a held create and exited 0 after 132978 ms; a GET held 100 s was 1 attempt, 103298 ms

functionPresent() in the same harness retries 429 only, with a fixed 15 s sleep (3 attempts in 30144 ms for two 429s), and a 500 on its read returns present: false with status 500, which a caller cannot tell from a missing function without checking status (MF01c). A held POST is not resent by either tool: the CLI exits 1 at 60 s (60448 ms for a 75 s hold) with the create already landed (MF05a), and the provider waited out the hold (MF05b). Writes answered 502, 503, 504 or 429 in the provider, PATCH through the CLI and connection-level faults were not run.

A create answered 500 after upstream had created the project (injected by the proxy forwarding the request and replacing the answer) leaves the client with an error and the project upstream. In the provider (MF04b-c), apply exited 1 with 1 POST, state held 0 resources, and a plain re-apply answered 400 (same name in the organisation) with the orphan still there; deleting the orphan through the API let the next apply exit 0 in 3468 ms. The CLI’s projects create answered 500 after upstream created it did the same, 1 attempt, exit 1, 1 project with the requested name (MF03). The same-name 400 is the only duplicate protection observed, and a retry with a fresh name is not covered and was not run (MF02c). A settings write that failed before upstream leaves the project in state without its settings (MF04d0); one that landed but answered 500 converges at the next plan (MF04d2); a DELETE that landed but answered 500 leaves a state entry that destroys cleanly later, because a 404 on the GET and the DELETE is accepted (MF04f2-f3). The harness create behaves the same way on an answer lost to its own 4 s timeout (MF02a, 1 project afterwards).

Restore paths (PITR, Free-plan pause and unpause, and what the Management API exposes for each) are priced and compared in the DR tiers reference, which owns that tier ladder.

Measured in http-tier-lockdown, a PostgREST schema-cache wedge returns 503 PGRST002 “schema cache load”; the fix is pg_notify('pgrst', 'reload schema'). PGRST001 on an empty exposed schema is not a wedge. ACTIVE_HEALTHY means the HTTP tier is up, not that PostgREST can reach Postgres - probe a real table, never the status field.

Class 6: Storage render path failure modes

Section titled “Class 6: Storage render path failure modes”

Measured in edge-resilience W19 (2026-08-16): all six outcomes, verbatim:

Source objectRender URL (?width=32)Plain public URL
Valid PNG200, resized200
corrupt.png (text bytes)400 InvalidRequest - “The source image is invalid or unsupported for rendering”200, original bytes
SVG200, source SVG unchanged (no rasterisation)200

The render path never produced a 5xx, and the original always serves, which the workaround relies on.

Pre-generate renditions at upload (the docs-recommended architecture; the billing fix doubles as a resilience fix against the render path). The per-project transformations toggle is a hard stop, though cached transformed images may still bill after disabling, so do not promise a clean zero.

The billing half is a commercial decision, not an incident class: transformations bill per distinct origin image per billing cycle ($5 per 1,000, 100 included on Pro and Team, count resets each cycle), a growing library re-bills every month it is viewed, and the cap trade-off - cap on means eventual restriction, cap off means an uncapped bill - is priced out in the DR tiers reference.8

Class 7: Edge caching as an outage absorber

Section titled “Class 7: Edge caching as an outage absorber”

Measured in edge-resilience W04 (2026-08-15), a cache-first Cloudflare Worker served warm URLs 200 with byte-identical bodies while the origin was hard-down. Cold URLs failed.

Cloudflare Workers wraps TCP failures to unroutable addresses as a 403 response rather than a JS exception, so catch-based stale fallback never fires for that failure mode; handle it in the status branch. The same wrap affects failover logic: a >=500-only failover condition never trips on an unroutable primary, because the failure arrives as a 403 response.

Measured in edge-resilience W24 (2026-08-16), the failover proxy ran end to end, every step HTTP 200: prime the probe URL (origin: primary) -> redeploy with the primary pointed at an unroutable address (origin: standby) -> restore the primary and probe immediately (origin: still standby - the HOLD_MS flap-damping window, persisted in the Cache API so it survives the redeploy) -> probe after the window (origin: primary). Three bugs, each measured, had to be fixed before the sequence ran green:

  1. Cache-first masks failover. The first iteration measured origin “none” on the prime and outage probes: the cache-first read ran before the failover logic, and HIT responses carry no origin information. A failover mode behind a cache-first read is masked by warm entries - skip the cache when failing over, and tag every response with the origin that served it.
  2. The 403 wrap defeats >=500 failover conditions (the wrap described above). The shipped worker fails over on 5xx, 403, or any non-ok in outage mode.
  3. PostgREST 400s on unknown query params (treated as column filters). The drill’s ?_w24= cache-buster, forwarded to the origin, 400’d every probe - and a 400 is not a failover condition, so the outage phase served 400s tagged primary. Strip probe-only params from the origin URL; keep them only in the cache key.

The hold window must exceed the redeploy-plus-settle path between the last outage probe (which refreshes the failure timestamp) and the holdover probe - ~11s observed, so HOLD_MS=15000 measured an expired window and 60000 is what the drill ships.

The multi-tenant routing variant of the drill worker (W25) isolates failures per tenant: a stale routing row pointed at a dead origin fails only that tenant (502 for it, 200 elsewhere), and the eject costs a redeploy (10.6s) while the table lives in an env var (measured, edge-resilience W25).

Custom hostname and a resolver that cannot resolve supabase.co

Section titled “Custom hostname and a resolver that cannot resolve supabase.co”

A hostname you own in front of the project is the client-side lever the platform itself used when it published a proxy domain during a regional block, which restored PostgREST, Realtime and Edge Functions while auth callbacks and storage object URLs stayed affected through hardcoded domain references.9 Measured in hostname-path (2026-10-10, one Pro project in ap-southeast-1, one APAC vantage, n=1 per row; the custom-domain add-on behaviour itself is in Static hosting on Supabase):

  • HP06 resolution: the project host is 2 A records, no AAAA, TTL 300, the same set through the system resolver, 1.1.1.1, 8.8.8.8, 9.9.9.9 and DoH on Cloudflare and Google. A custom hostname set as a CNAME (TTL 60) to <ref>.supabase.co returned the same 2 addresses on all six paths. supabase.co is delegated root, then co. (4 NS), then 2 NS on Cloudflare, with no DS at .co and no DNSKEY, and an unused label under it answers NXDOMAIN (no wildcard).
  • HP07 rig: a recursive Unbound in Docker with a zone forward sending supabase.co. to a dnsmasq that answers NXDOMAIN, and an app container whose only resolver is that Unbound. The app is supabase-js doing 6 operations (a DNS lookup, password sign-in, a REST read, a public Storage object, an Edge Function call and a Realtime subscription). Control, recursion intact: 6 of 6 ok on the custom host and on the project host.
  • With supabase.co NXDOMAIN and the custom host a CNAME to the project host, the app on the custom host did 0 of 6 operations (getaddrinfo ENOTFOUND), and so did the app on the project host. The CNAME is in the answer and its target is NXDOMAIN, so a custom hostname that is a CNAME to <ref>.supabase.co does not survive a resolver that cannot resolve supabase.co.
  • With the CNAME replaced by A records holding the project host’s 2 addresses, the same hostname answered NOERROR with 2 A records and the app did 6 of 6 operations. Before that, curl pinned to each address with SNI set to the custom host returned 200 on both. The platform status still read 5_services_reconfigured a few seconds after the CNAME was removed. How long the addresses stay valid (they are shared anycast addresses), a reverify or certificate renewal without the CNAME, and whether the platform supports address records were not tested; this records a measured behaviour and recommends nothing.
  • A resolver that had the chain cached before the outage kept answering at every 30 s sample to 271 s, and failed first at 301 s, matching the 300 s TTL on the cached target A record (one resolver configuration, no serve-stale and no prefetch; a stale-serving resolver would differ and was not run).

Not simulated: an outage seen by some resolvers and not others, a failure at the .co TLD (the custom hostname sits under a different TLD, so only the CNAME target is affected), and ISP-side filtering. The auth side of the same hostname (the callback moving to the custom host about 6 s after activation, and an access token iss that stays on the project host) is measured in HP03 and HP04; the auth mechanics are in Supabase Auth end to end.

Measured in edge-resilience W02 against supabase-js 2.112.3: 401 PGRST303 -> exactly 1 attempt (no retry amplification). 503 x3 then 200 -> 4 attempts, success in ~7.0s. Connection refused -> surfaced in ~7.0s. The built-in retries are on by default since v2.102.0; custom policies go through fetch-retry.10 Which statuses and methods they cover was measured on 2026-10-10 in client-retries (CR01-CR04): a local fault proxy sat between the client and the project’s REST URL, so an injected status never reached the platform and the proxy log, including X-Retry-Count, is the source of every attempt count. supabase-js 2.112.3 on Bun 1.3.14 unless a row names another version or Node v26.11.1; n=1 per case unless stated.

Answered by the serverGET, HEAD, rpc({ get: true })POST insert, PATCH, DELETE, upsert, default rpc()
5034 attempts at 1, 2, 4 s, X-Retry-Count 1, 2, 3 (7008 ms; CR01a2, CR01b3)1 attempt (CR01b1-b3)
5204 attempts, 7012 ms (CR01a3)1 attempt (CR01b1)
Connection reset4 client attempts, 7016 ms (CR01a5)1 attempt (CR01b1)
525, 502, 504, 500, 429, 408, 521, 522, 5241 attempt for each of the nine (CR01a4)not run per status; the method is never retried
409not in the measured setnot run
401 PGRST3031 attempt (W02)not run

A GET reset gave 4 client attempts but 5 requests on the wire on the shared proxy in CR01a5 (the first request sent twice with no header change), and 4 wire requests in every version in CR03, where each case had a fresh mock server; the cause of the extra request was not isolated. The 503 row holds with Retry-After unset. With it, the header sets the spacing: Retry-After: 2 gave a 2000 ms gap, 0 gave 0 ms, 40 gave 40000 ms (above the 30 s cap), and an HTTP date was ignored with a 0 ms gap (CR01a6). The real Data-API-off 503 (PGRST002, with the schema list emptied through the postgrest config endpoint of the Management API) carries Retry-After: 0 on every answer, so the default client’s four attempts ran back to back (gaps of 1700, 1800 and 2000 ms, each about the previous answer’s duration plus 40 ms, 7252 ms in total) rather than at 1, 2, 4 s; a POST made 1 attempt and a GET with .retry(false) 1 (CR01e1-e3). The gateway’s edge_logs showed 5 rows for those calls and no X-Retry-Count (source = 'edge_logs'), so attempts are visible there only as rows (CR01f).

supabase.functions.invoke is a neighbour of this class and does not share the policy: supabase-js 2.117.3 made 1 attempt and the function received 1 request for an injected 503 (with and without Retry-After), 502, 500 and 429, 3 trials per status, error class FunctionsHttpError (edge-runtime-auth ER02-retry-503). A 503 produced by the platform’s own relay, network errors and other client versions were not measured. Limits of the function side are in Edge Function limits, one ceiling at a time.

Per version, from a local mock with a fresh server per case (CR03, both runtimes), cells are attempts:

supabase-jsGET 503GET 525POST 503.retry(false)db: { retry: false }db: { retryEnabled: false }
2.101.1111111
2.102.0 to 2.111.0 (10 versions)411144
2.112.0, 2.112.3, 2.117.3411114

Retries begin at 2.102.0. db: { retry: false } is honoured from 2.112.0, so on 2.102.0 to 2.111.0 only .retry(false) on the query opts out. db.retryEnabled, the name the changelog gives, changed nothing in any of the 14 versions. A per-query .retry(true) beats db.retry = false (4 attempts, CR01c4).

  • There is no default timeout: a response held 10 s completed (10108 ms, CR01d2), and a request the server accepted and never answered was still pending when observation stopped at 330004 ms on Bun and 330016 ms on Node (CR04, one trial each, default client, no db.timeout, no signal).
  • db.timeout is per attempt, not per call: with 2000 and a 503 on every attempt the call made 4 attempts and took 7010 ms (CR01d4). An abort is not retried: a response held 3 s with db.timeout = 2000 ended at 2004 ms after 1 attempt (CR01d3).
  • Under Bun 1.3.14, the docs page’s own pattern of .abortSignal(AbortSignal.timeout(n)) created inline did not end a call that was sleeping between retries: a 5000 ms deadline with a 503 on every attempt ran 4 attempts to 7011 ms (CR01d5a), and the same with 2500 ms ran 4 attempts in about 7 s on every version from 2.102.0 (CR03). Node ended after 2 attempts at about 2.5 s. An AbortController with setTimeout, or a timeout signal with an abort listener attached at creation, ended at the deadline on both runtimes (CR01d5b and CR01d5c, 5005 ms and 5003 ms with 3 attempts). The cause is not isolated and no Bun fix or other Bun version was checked. A plain fetch with an inline AbortSignal.timeout(2000) against a held request aborted at 2002 ms (a scratch check, Bun and Node).

CR02 ran two client policies on Bun. A deadline plus one hedged GET (built-in retries off, one deadline signal with a listener, one extra request when the first fails with status 0 or 5xx or has been outstanding 1500 ms) bounded a held request to 1575 ms (first request held 10 s, deadline 5000 ms), and a request never answered to the 4000 ms deadline (ended at 4003 ms, 2 requests, both aborted). It sent 2 requests for a 525 where the built-in policy sends 1. Left on under the hedge, the built-in retries sent 6 requests in 6 s against 2 with them off (CR02f). The hedge is for GET only: a hedged insert into a table with no unique key wrote 2 rows for one logical write, and a hedged primary-key upsert wrote 1 (CR02g, the server made slow rather than the network). The policy retries immediately; a pause or jitter was not tested, and neither was retry load on the Data API connection pool, which the docs warn about.10

Refresh-then-retry on a 401 (one auth.refreshSession() and one repeat) recovered an injected 401 PGRST303 with 1 refresh call and 2 REST requests (200 after 136 ms), and a real 401 from a corrupted access-token signature (PGRST301) the same way; 401 on every request stopped after 1 refresh and 1 retry; a session revoked with POST /auth/v1/logout?scope=global returned the original 401 after a failed refresh (refresh_token_not_found); five parallel requests each answered 401 once made 1 refresh call (CR02h, one trial, so client deduplication cannot be told from timing luck). The injected 401s are not the platform’s: the incident class “fresh JWT rejected” was not reproduced.

Neither public page lists 408, 409 or 504. This page’s earlier text said the built-in set was 408/409/503/504; 408 and 504 each got 1 attempt on a GET (CR01a4) and 409 was not measured. The docs page lists GET, HEAD and OPTIONS (OPTIONS was not tested) with 503, 520 and network failures, and names the opt-out db: { retry: false }; the changelog says GET and HEAD, up to 3 retries at 1, 2, 4 s capped at 30 s, and names retryEnabled.1011 The spacing, the methods and the version match; retryEnabled does nothing and db.retry needs 2.112.0 (above). Neither page, as read, mentions Retry-After. Filed upstream: no.

Class 9: Warm standby and cutover (the HA tier)

Section titled “Class 9: Warm standby and cutover (the HA tier)”

The auth findings in this class and in classes 1 and 8 are consolidated, with the refresh, third-party-issuer and self-hosted GoTrue measurements that postdate them, in Supabase Auth end to end.

The full HA path works on managed projects (edge-resilience W05, 2026-08-15, cross-region ap-southeast-2 -> ap-southeast-1):

  • Logical replication works managed to managed, with a subscription on the standby against the primary’s direct host (db.<ref>.supabase.co). Initial sync ~3.1-6.5s on a small table; replication lag 34ms-1057ms across regions. The pooler cannot be the source - it fails at the tenant-identifier layer (ENOIDENTIFIER).
  • Sessions survive cutover without copying secrets once the primary’s OIDC issuer is registered as a TPA integration on the standby (resolves in ~60-122ms); primary-issued tokens then read the standby’s API. Copying the JWT secret is not an option anyway - the config API accepts the write and changes nothing (measured 2026-08-14).
  • A first-time issuer’s key costs ~30s of PGRST301 before PostgREST trusts it; a previously-seen JWKS warms in ~300ms. Rehearse the cutover before you need it.

A standby replicates your mistakes too: a bad DELETE lands on the standby in 34ms, so recovering from logical corruption needs PITR.

Cutover hygiene, all measured (2026-08-16 unless noted): sequences do not replicate - the first post-cutover insert fails with a verbatim duplicate-key error (duplicate key value violates unique constraint "w16_t_pkey"), and setval resync restores inserts (W16). DDL does not replicate, and primary-side DDL stalls ALL table replication - even rows not using the new column, because the apply worker holds streaming changes for any table not yet in r state - until the same DDL lands on the standby, which resumes replication in ~6.1s with backfill and no subscription recreation; apply standby DDL first (W15). Per-project auth config (SMTP, SITE_URL, redirect URLs, rate limits, jwt_exp) does not follow a cutover - re-apply it via the Management API; the diff is readable verbatim from GET /v1/projects/{ref}/config/auth on both projects (W17). In Realtime, dropping and recreating a table under an active subscription kills events for that table NAME (new OID, stale channel metadata) - stable canary tables only (W12). And the wedged-subscription recovery is ordered: disable -> set (slot_name = none) -> drop subscription -> pg_drop_replication_slot on the publisher - dropping without the slot detach leaves the publisher slot pinning WAL.

W09 and W14 (2026-08-15/16) located the replication boundary. Sizing the standby up ruled out worker exhaustion on micro: pg_settings DO scale with size (max_connections 60 -> 90, shared_buffers 256MB -> 512MB) but max_worker_processes is 6 on both micro and small - platform-fixed. The failure differs by mode: with copy_data=true the initial sync stalls in pg_subscription_rel state d forever, the sync worker hung at IPC/BgworkerStartup with its publisher sync slot inactive; with copy_data=false the WAL sender connects but received_lsn stays NULL - zero changes stream either way, at any size. A custom non-public schema (lab_schema.t) replicates in ~4s on the same instances, while storage.buckets does not - and inserts via the postgres role change nothing, so the writer role is not the filter. The boundary is specific to platform-managed schemas (auth.*, storage.*); size, worker count and schema privacy were each ruled out. The standby auth options are TPA token portability for existing sessions, user backfill through the admin API or direct SQL, or forced re-login for fresh logins. POST /auth/v1/admin/users accepts the source’s bcrypt password_hash and the user keeps their password (measured, tenant-consolidation C03, 2026-08-04); copying hashes via direct SQL into auth.users is the other path (not measured on a standby). The W09 run (2026-08-15) posted its backfill users without a hash, so its “not portable via the admin API” note was an assumption; C03 measured the hash path.

A 1,000,000-row initial sync (W22) completed in 22.7s (12.5s on a later battery run), with streaming lag ~245-276ms immediately after, so the small-table lag figures still hold at that size, and the sync scales without the stall the earlier runs suggested.

The cold DR floor (W06) is pg_dump 12.4s and restore 6.4s for 10k rows through the pooler session host.

For break-glass access (W07), GET /v1/projects/{ref}/postgrest returns the project’s jwt_secret, and a locally minted token authenticates against the live API with zero Auth involvement. During an Auth outage that keeps the API usable; the same read grants full token minting for the project. Prefer TPA portability - and rotate the secret after any break-glass use: the Management API read is auditable, but the secret stays live until rotated.

Measured in edge-resilience W13/W18 (2026-08-15/16), the wall clock is a 150s idle timeout - a sleeper function holding the connection past it dies with 504 IDLE_TIMEOUT (W13). Cold start costs ~1.4s on the first invoke after deploy plus idle; subsequent idle-cold invokes land at 121-302ms. Over 5 cold invokes at 60s gaps: p50 284ms, p99 1433ms; warm p50 over 20 invokes is 98ms (W18). Keep-warm pings recover the difference between 98ms and ~300ms and do not remove the 1.4s; long jobs belong on a queue plus cron rather than in a request path. The deploy-time ceilings (functions per project, bundle size by deploy path, the four secrets limits) and the failures that only look like limits (silent loss under parallel deploys, the 413 that both size ceilings return) are measured separately in Edge Function limits, one ceiling at a time.

Class 11: Statement and lock timeout signatures

Section titled “Class 11: Statement and lock timeout signatures”

Measured in edge-resilience W20 (2026-08-16), both surfaces return HTTP 400, and the SQLSTATE in the body is what tells them apart:

  • statement_timeout (2s setting against pg_sleep(5)): verbatim ERROR 57014: canceling statement due to statement timeout, 3467ms wall.
  • lock_timeout (3s setting on an advisory-lock contender, two sessions): verbatim ERROR 55P03: canceling statement due to lock timeout, 4533ms wall.

Alert on the codes; a 400 alone is ambiguous with ordinary bad queries.

A lock timeout raised by a schema change rather than by a query you wrote has its own shape, and the lock you asked for decides whether it happens at all: a validating ADD CONSTRAINT on a table that references auth.users holds SHARE ROW EXCLUSIVE on auth.users and blocks sign-in until it commits, while NOT VALID plus VALIDATE holds only ROW SHARE during validation (the locks, and the two-statement path).

Realtime limits are per plan (documented, 2026): Free 200 concurrent connections / 100 messages per second; Pro 500 / 500; Pro without the spend cap and Team 10,000 / 2,500; Enterprise beyond that.12 Exceeding the message rate disconnects clients with a tenant_events error, and supabase-js reconnects automatically once throughput drops below the plan limit. It looks like an outage, with sockets dropping en masse; the disconnect reason identifies it as a quota, and the fix is plan or usage.

Fleet-wide platform incidents have no client-side answer - the remedy is contractual (SLA, degradation prioritisation). The platform uptime SLA is Enterprise-only - 99.9% per product per month, measured per-project, per-region or globally depending on the product, GA features only, with service credits of 10-30% of the affected service’s monthly fees (capped at 20% of the trailing twelve months’ fees) as the sole remedy.13 Support SLAs exist for Team and Enterprise: Urgent response is 24 hours 24/7x365 on Team and 1 hour 24/7x365 on Enterprise Standard and Priority Plus; business-hours limits start at High (Team: 1 business day for High and Normal, 2 business days for Low). The full severity table is on the SLA page.13 Everyone below Team is on best-effort. Read replicas are read-only (Data API GET, or SQL over the replica’s pooler), have no documented promotion path, and Auth always goes to the Primary, so they absorb read load only.14 Multi-region active-active writes risk split-brain: dual-write returns 200/200 with ~107ms skew, but a partial failure (primary 200 / standby 400) is not atomic (W26), so fail over rather than dual-write.

One row per practice, with the number or error it addresses and the module it rests on. Module ids resolve in the edge-resilience RUNLOG, the platform-downtime RUNLOG, the instance-sizing RUNLOG and the compute-disk RUNLOG. The platform-downtime rows are n=1 per operation, so they rest on the ordering of the paths, not on the seconds. The 2026-10-10 rows resolve in the client-retries RUNLOG, the pooler-checkout RUNLOG, the lifecycle-ops RUNLOG, the mgmt-api-faults RUNLOG, the hostname-path RUNLOG and the edge-runtime-auth RUNLOG; they are n=1 per cell unless the row gives a count, and the control-plane rows are injected faults.

PracticeEvidenceModule
Size a maintenance window off the longer of the restart and resize figures.Auth 131 s and the pooler 207 s resizing up, against 75 s and 158 s for a restart (2026-08-04); on 2026-10-23 Auth was down 247 and 192 s restarting against 171 and 218 s resizing Micro to Small.platform-downtime D01 vs D03, 2026-08-04
Do not count on PostgREST or Realtime to ride out a planned restart or resize.Row reads and Postgres Changes joins failed for about as long as Auth and Storage: restart 247 and 194 s, Micro to Small 171 and 218 s, Medium to Large 113 s for the read (a second run is void). The 2026-08-04 “zero failed samples” was an anon GET /rest/v1/ the gateway answers with 401, and a WebSocket handshake.platform-downtime D01, D03, D04, 2026-10-23
Read (EADDRNOTALLOWED) from the pooler as a restriction; the other strings do not name the operation.On 2026-08-04 {:error, :timeout} was a restart, {:error, :econnrefused} a resize up, terminating connection due to administrator command a resize down, (EADDRNOTALLOWED) address not in tenant allow_list a restriction just applied; on 2026-10-23 restarts and resizes up both answered terminating connection due to administrator command.platform-downtime pooler-mode table
Allow-list your pooler clients’ egress addresses as well as your direct ones.A network restriction is enforced by Supavisor against the client address, so 6543 is covered and the refusal lands 1 s after sampling starts.platform-downtime D02, two runs
Retry the first write after ACTIVE_HEALTHY, and the first Storage calls on a fresh project.5 of 5 fresh projects refused the first POST /auth/v1/admin/users with 500 unexpected_failure and accepted the next attempt one poll tick later. Fresh-project Storage answers TenantNotFound and then 429 SlowDown for the first minutes. A pre-provisioned project pool (Class 4) needs both.placement reference, Verified row ‘Create -> healthy, and healthy is not writable’ (2026-08-04, n=5, bash run not in the lab repo); supabase-lab AGENTS.md provisioning note (2026-08-03, 2 of 2, 500 "Database error checking email"); W21
Pass a smart region group on POST /v1/projects when capacity matters more than the exact city.region_selection: {"type": "smartGroup", "code": "apac"} was accepted on a Pro org with 201, landed ap-northeast-2, healthy in 135 s.instance-sizing I02, 2026-08-17
Retry Realtime joins with backoff, and never drop and recreate a table under an active subscription.A join after 11 other modules had the socket closed before the event arrived. After a drop and recreate, events for that table name stop (new OID, stale channel metadata).W12
Keep the tenant routing table in KV or D1 rather than an env var.The env-var eject cost a redeploy at 10.6 s. The KV/D1 eject time itself is not measured.W25, 2026-08-17; the KV/D1 half is a design choice
Diff the whole config/auth payload between primary and standby rather than a named field list.On GET /v1/projects/{ref}/config/auth, custom_oauth_max_providers defaulted to 32767 on one project generation and 3 on another.W17 baseline diff, 2026-08-17
Poll the readback after any config/auth PATCH, including the restore in a drill.The apply is async and an immediate readback returned the pre-restore value.W17, 2026-08-17 battery (24/25, green on re-probe)
Check max_replication_slots before adding a subscription and drop orphan slots on the publisher first.The limit is 10 on micro and on small. Each subscription pins one, and a dropped subscription can leave its slot on the publisher either way - after a plain drop (W05) and after slot_name = none then drop (W14) - so list pg_replication_slots on the publisher and pg_drop_replication_slot any orphan.compute-disk D01; W05, W14
Add 504 and 408 to the GET retry set through fetch-retry; 520 is already retried.The built-in GET, HEAD and rpc({ get: true }) retries cover 503, 520 and connection resets, 4 attempts at 1, 2, 4 s. 504, 408, 525, 502, 500, 429, 521, 522 and 524 each got 1 attempt (injected, supabase-js 2.112.3, 2026-10-10). This row’s earlier text, 408/409/503/504, was not what either docs page says1011; 409 was not measured.CR01, 2026-10-10
Opt out with .retry(false); db: { retry: false } needs supabase-js 2.112.0..retry(false) gave 1 attempt from 2.102.0; db: { retry: false } was ignored through 2.111.0; db: { retryEnabled: false }, the changelog’s name, did nothing in 14 versions.CR01, CR03
Bound every call with an AbortController and setTimeout, not an inline signal.There is no default timeout (still pending at 330 s on Bun and Node, n=1 each) and db.timeout is per attempt (4 attempts, 7010 ms at 2000). On Bun 1.3.14 an inline AbortSignal.timeout did not end a call sleeping between retries (4 attempts, about 7 s for a 2.5 s or 5 s deadline); Node did.CR01 (d4, d5), CR03, CR04
Hedge only GETs, with the built-in retries off under the hedge.A hedged GET bounded a held request to 1575 ms; with the built-in retries left on it sent 6 requests in 6 s against 2. A hedged plain insert wrote 2 rows, a hedged primary-key upsert 1.CR02 (f, g), Bun only
Write the retry for functions.invoke yourself; the client makes 1 attempt.1 client attempt and 1 function-side request on an injected 503, 502, 500 or 429, 3 trials per status, supabase-js 2.117.3. A platform-relay 503 was not measured.ER02 (ER02-retry-503)
Send the first statement on a full transaction pool through 5432 session mode.Client 17 connects in 43 to 52 ms and its first statement fails at 60,007 to 60,009 ms (6 of 6); the session pooler answered 49 to 70 ms after the primary’s failure and as a 17th backend. Direct 5432 is ENOTFOUND for an IPv4-only client without the add-on.PC01, PC02, 2026-10-10
Wait longer than the outage before the one retry of a stale pooled connection.After a 150 ms flap a retry at 0 ms rescued 0 of 30 on node-postgres and on Prisma; at 300 ms it rescued 25 of 25 and 30 of 30.PC03
Attach an 'error' listener to every node-postgres Pool.A Pool without one exited non-zero in 9 of 12 runs at a 2 s cut; with one, 0 of 36 runs crashed.PC03
Read db_host from the pooler config endpoint, not a literal aws-0 host.Fresh Pro projects in two regions reported aws-0 and the aws-1 host answered ENOTFOUND tenant/user. Older projects and other regions were not measured.PC04
Look a project up by name before retrying a create, and keep the name stable.An abandoned create still created the project and an identical re-send answered 400 already exists (n=1). A provider create answered 500 after upstream created it left 0 resources in state, an orphan, and a 400 on every re-apply until the orphan was deleted. A fresh-name retry was not run.LO04, MF02, MF04
Gate lifecycle calls on the worst status over every component named for the region.189 components carry 18 region names 10 or 11 times each with no group field, incidents.json has no region field, and impact was none on incidents that suspended upgrades. A replayed restart gate would have refused ap-southeast-1 for 38.5 hours of 1113 (heuristic).LO01, 2026-10-10
Do not read control-plane ACTIVE_HEALTHY as the end of a restart outage.Status was back at 40 to 71 s while the pooler ran to 151 s in one of 5 restarts; Auth 54 s and Storage 58 s at p50 (2026-10-10). In the 2026-10-23 re-run the status read healthy at 172, 142, 41, 41 and 141 s, each time before the REST read recovered (windows 181, 154, 46, 46 and 150 s from a first failure 1 to 3 s after the POST).LO05; LO05 2026-10-23 artifact
Do not rely on a CNAME to the project host as the only path through a DNS outage.With supabase.co NXDOMAIN at the resolver a CNAME custom hostname did 0 of 6 operations; A records holding the 2 project addresses did 6 of 6. A resolver with the chain cached failed first at 301 s (300 s TTL). Certificate renewal without the CNAME was not tested.HP07, 2026-10-10
Cap Management API calls yourself; the CLI waits 360 s on a stalled GET.The CLI retries GET, PUT and DELETE 6 times with no pause on a 500 and POST once, 60 s per attempt (a held GET took 360220 ms); the provider retries GET only; the harness mgmt() never retries. Every fault was injected; no real 5xx was seen.MF03, MF04

Not measured: the KV/D1 eject time (W25 measured the env-var redeploy path only); whether a primary-side network restriction blocks a standby’s walreceiver (D02 restricted the pooler, the direct-host subscription path was never restricted); failing back by reversing the replication direction; the 12h jwt_exp value (the lever is measured, the value is a choice). Read replicas kept GETs answering, stale, while the primary restarted, but not while the replica did: the restart took both nodes down one after the other, and while the replica was down the load balancer failed every read and sent none to the primary (replica-routing RR02i, 2026-10-23, two runs; see read replicas). The discrepancy this page used to carry between compute-disk D09 (0 s on /auth/v1/health across four resizes, 2026-08-19) and platform-downtime D03 (131 s of HTTP 521, 2026-08-04) is resolved: the old D09 probe sent no apikey and dropped outages still open at a phase end, and its 2026-10-23 re-run with the anon key (250 ms sampling) saw /auth/v1/health down 152.375 to 254.869 s per resize, n=1 per op (compute-disk RUNLOG). No module has probed an authenticated Auth operation across a resize.

What is failing?Reads 5xx / origin down401 PGRST303 waveLogins / refresh deadProject or region lostCache-first Worker (W04)Raise jwt_exp (W03)+ alert on codeLonger TTL:existing tokens still validateWarm standby + TPA cutover (W05)rehearsed

Text version of the decision guide, keyed on what is failing:

  1. Reads 5xx or the origin down -> a cache-first Worker (W04).
  2. A 401 PGRST303 wave -> raise jwt_exp (W03) and alert on the code.
  3. Logins or refresh dead -> a longer TTL, where existing tokens still validate.
  4. Project or region lost -> a warm standby and TPA cutover (W05), rehearsed.
ClaimValueHow it was checked
Skew tolerance boundary+30s pass, +31s PGRST303Measured (W01, two runs)
jwt_exp acceptance-to-effect~6.5sMeasured (W03)
supabase-js on PGRST3031 attempt (supabase-js 2.112.3)Measured (W02, mock; the other statuses are the CR rows below)
Built-in retries, GET, HEAD, rpc({ get: true })503, 520, reset: 4 attempts at 1, 2, 4 s; 525, 502, 504, 500, 429, 408, 521, 522, 524: 1Measured, injected statuses on a local proxy, supabase-js 2.112.3, Bun 1.3.14 (CR01, 2026-10-10, n=1 per case); docs: 503, 520, network failures, not 408, 409 or 5041011; filed upstream: no
Built-in retries, POST, PATCH, DELETE, upsert, default rpc()1 attempt on 503, 520 and resetMeasured (CR01)
Retry-After on a 5032 -> 2000 ms gap, 0 -> 0, 40 -> 40000 ms (above the 30 s cap), HTTP date ignored; real Data-API-off 503 sends 0, 4 attempts in 7252 msMeasured (CR01)
Opt-out names by version.retry(false) from 2.102.0; db: { retry: false } from 2.112.0; db.retryEnabled none of 14 versionsMeasured (CR03, Bun and Node)
Default timeoutnone: pending at 330004 ms on Bun, 330016 ms on Node; db.timeout per attemptMeasured (CR04, one trial each; CR01)
Inline AbortSignal.timeout between retriesBun 1.3.14: 4 attempts, about 7 s for a 2.5 s or 5 s deadline; Node: 2 attempts at about 2.5 sMeasured (CR01, CR03)
Deadline plus one hedged GETheld request 1575 ms, hang 4003 ms at a 4000 ms deadline; built-in retries on under the hedge 6 requests in 6 s vs 2; hedged insert 2 rows, hedged PK upsert 1Measured (CR02, Bun only)
Refresh-then-retry on a 4011 refresh call and 2 REST requests; 5 parallel 401s made 1 refreshMeasured (CR02; the parallel case is one trial)
functions.invoke on 503, 502, 500, 4291 client attempt, 1 function-side request, 3 trials per statusMeasured (ER02, supabase-js 2.117.3, injected)
Edge cache under origin outage200, byte-identicalMeasured (W04, forced outage)
Standby replication lag34ms-1057msMeasured (W05, three runs)
Standby initial sync, 1M rows22.7s / 12.5s; lag after ~245-276msMeasured (W22, two runs)
auth./storage. replicationzero changes at any tested sizeMeasured (W09/W14)
DDL on primary stalls all table replicationresume ~6.1s after standby DDLMeasured (W15)
Post-cutover insert without resyncduplicate key; setval fixesMeasured (W16)
Edge function cold start~1.4s first-after-idle only; p50 284ms cold vs 98ms warmMeasured (W18)
Edge function wall clock150s idle -> 504 IDLE_TIMEOUTMeasured (W13)
Render path on invalid source400 InvalidRequest; original still servesMeasured (W19)
statement_timeout signature57014 verbatim, 3467ms wallMeasured (W20)
lock_timeout signature55P03 verbatim, 4533ms wallMeasured (W20)
pg_cron across restartresumes on schedule (2 -> 5 rows), no doublingMeasured (W23)
Edge failover sequenceprimary -> standby -> holdover -> primary, all 200Measured (W24)
Tenant routing isolationpoisoned row: 502 for its tenant, 200 elsewhere; eject = redeploy (10.6s) via env-var tableMeasured (W25)
Storage dual-writeparallel 200/200, ~107ms skew, bytes equal; partial failure not atomic; sync-after 97msMeasured (W26)
Spend cap enforcementnone at the request path (quota+5 renders all 200)Measured (W21)
TPA resolution on standby~60-122msMeasured (W05)
Cold kid trust on cutover~30s PGRST301Measured (W05 full suite)
Cold DR, 10k rowsdump 12.4s, restore 6.4sMeasured (W06)
Break-glass minting works200 real / 401 wrong secretMeasured (W07)
Concurrent refresh raceboth 200Measured (W08)
Restart gaps REST (HTTP tier)/Realtime (handshake)/Auth/Storage/pooler0 failed samples / 0 / 75 s / 78 s / 158 sMeasured (platform-downtime D01, 2026-08-04, n=1, 500 ms); RELIABILITY.md carried ~10s / ~26s / ~75s (N=3) with no artifact; re-checked 2026-10-10 in the next row
Restart re-check REST (HTTP tier)/Auth/Storage/pooler/direct (Micro, ap-southeast-1, n=5)0 failed runs / 54 s / 58 s / 54 s / 50 s window p50; max 60 / 59 / 151 / 56 s; first failure 2 to 3 s after the POST; control plane ACTIVE_HEALTHY at 40 to 71 sMeasured (LO05, 2026-10-10, 0.5 s sampling, Realtime not sampled)
Restart, data path: REST read/Realtime join (Micro, ap-southeast-1, n=5)failed 5 of 5; window p50 150 s / 146 s, max 181 s / 179 s; HTTP-tier probes 0 of 5Measured (LO05 re-run, 2026-10-23, 0.5 s sampling)
Restart refused after a create400 about 70 s after the creating POST; accepted 300 s after the first ACTIVE_HEALTHY read on another projectMeasured (LO05, two projects, boundary unresolved)
Resize up Auth/Storage/pooler131 s / 127 s / 207 sMeasured (platform-downtime D03, 2026-08-04, n=1)
Fresh project: first write after ACTIVE_HEALTHY5 of 5 refused with 500, next attempt acceptedMeasured elsewhere (placement reference, Verified row ‘Create -> healthy, and healthy is not writable’ (2026-08-04, n=5, bash run not in the lab repo); supabase-lab AGENTS.md provisioning note (2026-08-03, 2 of 2, 500 "Database error checking email"); summarised in operation cost)
Smart-group region selection on a Pro org201, landed ap-northeast-2, healthy in 135 sMeasured (instance-sizing I02, 2026-08-17)
max_replication_slots on micro and small10 / 10Measured (compute-disk D01)
Pooler checkout with 16 backends held (Micro)client 17 connects in 43 to 52 ms; first statement fails at 60,007 to 60,009 ms with ECHECKOUTTIMEOUT (6 of 6); a statement queued under 60 s is servedMeasured (PC01, 2026-10-10; a public status mirror quotes 15000ms for other clusters, not reproduced)
Fallback to the session pooler on 543249 to 70 ms after the primary’s failure, 56 to 117 ms after the 60 s error; direct 5432 ENOTFOUND for IPv4-only without the add-onMeasured (PC02)
Driver pools after a TCP reset0 failed first attempts after the fault cleared (111 runs); retry-once at 0 ms 0 of 30 after a 150 ms flap, at 300 ms 25 of 25 and 30 of 30; Pool without an 'error' listener exited in 9 of 12 runsMeasured (PC03)
Pooler host prefixAPI reported aws-0 on fresh projects in two regions; aws-1 answers ENOTFOUND tenant/userMeasured (PC04, four projects)
Status-page shape and change-gate replay189 components, 18 region names 10 or 11 times each, no group field; impact ignored; replay would refuse ap-southeast-1 for 38.5 h of 1113Measured (LO01, one fetch at 00:43 UTC; the replay is a heuristic)
Create: bad bodies and ACTIVE_HEALTHY timing400, 404, 400, 400 in 55 to 194 ms, nothing created; healthy at 2 s twice and 145 s onceMeasured (LO03, n=3)
Abandoned create re-sentproject created; identical re-send 400 already existsMeasured (LO04, n=1)
supabase CLI 2.120.0 retriesGET, PUT, DELETE 6 attempts on 500 with no pause; POST 1 on 500, 502, 503, 504, 429; 60 s per attemptMeasured (MF03, injected faults, Pro org, ap-southeast-1)
OpenTofu provider 1.11.0 retriesGET 4 attempts on a persistent 503, 2 on one 500 or 429; POST, PATCH, DELETE 1 on 500; a 130 s held create exited 0Measured (MF04, MF05, injected faults)
Lab harness mgmt()1 attempt on 500, 502, 503, 504; TimeoutError at 30001 msMeasured (MF01, injected faults)
Create answered 500 after upstream created itprovider state 0 resources, 1 orphan, re-apply 400 until deleted; CLI exit 1 with 1 projectMeasured (MF02, MF03, MF04, injected)
Custom hostname as a CNAME with supabase.co NXDOMAIN at the resolver0 of 6 operations (ENOTFOUND); A records holding the 2 project addresses 6 of 6; cached chain failed first at 301 sMeasured (HP07, HP06, one resolver rig, n=1)
30s documented tolerancedocumentedPostgREST v11 docs1
Transform billing model$5/1000 distinct origins/cycleSupabase docs8

All measured rows come from the supabase-lab pvlab harness (experiments/edge-resilience), 2026-08-15/16/17, and are reproducible from the experiment’s Makefile - the full run log with every number is RUNLOG.md. Batteries: 2026-08-16, 22 modules, 22/22 (out/2026-08-16); 2026-08-17, 25 modules, 24/25 with W17 green on re-probe after its readback race was fixed (out/2026-08-17). The platform-downtime, instance-sizing and compute-disk rows cite their RUNLOGs at the same commit, and none of the modules they cite here has a published artifact: compute-disk has no out/ directory, and the platform-downtime and instance-sizing out/ directories hold D05 and I04 only, neither of which this page cites. The client-retries, pooler-checkout, lifecycle-ops, mgmt-api-faults, hostname-path and edge-runtime-auth rows are dated 2026-10-10 and are quoted by hand from their RUNLOGs. Their run artifacts are published redacted under each experiment’s out/ directory and linked per module in the Modules table.

ModuleExperimentTestArtifact
C01rls-wire-claimsc01-wire-claims.tsnone published
C03tenant-consolidationc03-users-merge-api.tsnone published
CR01client-retriescr01-retry-matrix.tsrun-2026-10-10T04-55-54-303Z
CR02client-retriescr02-client-policy.tsrun-2026-10-10T04-55-54-303Z
CR03client-retriescr03-version-matrix.tsrun-2026-10-10T05-09-44-295Z
CR04client-retriescr04-default-timeout.tsrun-2026-10-10T05-09-44-295Z
D01compute-diskd01-compute-pg-limits.tsnone published
D01platform-downtimed01-restart.tsnone published
D02platform-downtimed02-restriction-flip.tsnone published
D03platform-downtimed03-resize-up.tsnone published
D04platform-downtimed04-resize-down.tsnone published
D09compute-diskd09-compute-resize-timing.ts (2026-08-19); 2026-10-23 versionrun-2026-10-23T07-56-13-302Z
ER02edge-runtime-auther02-canary-drift-retry-latency.tsrun-2026-10-10T00-54-40-928Z-er02
HP03hostname-pathhp03-domain-up.tsrun-2026-10-10T04-28-53-141Z, run-2026-10-10T04-41-47-267Z
HP04hostname-pathhp04-oauth-custom.tsrun-2026-10-10T04-28-53-141Z, run-2026-10-10T04-33-23-282Z, run-2026-10-10T04-41-47-267Z
HP06hostname-pathhp06-resolution.tsrun-2026-10-10T04-34-30-810Z
HP07hostname-pathhp07-nxdomain.tsrun-2026-10-10T04-34-30-810Z
I02instance-sizingi02-smart-region.tsnone published
LO01lifecycle-opslo01-status-gate.tsrun-2026-10-10T00-43-34-355Z
LO03lifecycle-opslo03-create-fallback.tsrun-2026-10-10T00-36-15-870Z
LO04lifecycle-opslo04-duplicate-post.tsrun-2026-10-10T00-43-31-013Z
LO05lifecycle-opslo05-restart-envelope.tsrun-2026-10-10T00-43-31-013Z, run-2026-10-10T00-46-09-369Z
MF01mgmt-api-faultsmf01-harness-client.tsrun-2026-10-10T07-44-43-736Z
MF02mgmt-api-faultsmf02-harness-create.tsrun-2026-10-10T07-44-43-736Z
MF03mgmt-api-faultsmf03-cli.tsrun-2026-10-10T07-44-43-736Z
MF04mgmt-api-faultsmf04-tofu.tsrun-2026-10-10T07-44-43-736Z
MF05mgmt-api-faultsmf05-post-timeout.tsrun-2026-10-10T07-55-39-410Z
PC01pooler-checkoutpc01-checkout-timeout.tsrun-2026-10-10T01-53-22-718Z, run-2026-10-10T02-40-11-359Z
PC02pooler-checkoutpc02-fallback.tsrun-2026-10-10T02-40-11-359Z, run-2026-10-10T03-09-14-920Z
PC03pooler-checkoutpc03-driver-reset.tsrun-2026-10-10T01-53-22-718Z, run-2026-10-10T02-40-11-359Z, run-2026-10-10T03-20-56-512Z, run-2026-10-10T03-33-47-691Z
PC04pooler-checkoutpc04-pooler-host-lint.tsrun-2026-10-10T01-53-22-718Z, run-2026-10-10T02-40-11-359Z, run-2026-10-10T03-33-09-220Z, run-2026-10-10T03-46-45-971Z
PC05pooler-checkoutpc05-checkout-matrix.tsrun-2026-10-23T08-11-43-542Z, run-2026-10-23T08-11-48-633Z
PC06pooler-checkoutpc06-login-delay.tsrun-2026-10-23T08-11-43-542Z, run-2026-10-23T08-11-48-633Z
PC07pooler-checkoutpc07-pool-size.tsrun-2026-10-23T08-11-43-542Z, run-2026-10-23T08-11-48-633Z
W01edge-resiliencew01-jwt-skew.tsout/2026-08-16, out/2026-08-17
W02edge-resiliencew02-retry-probe.tsout/2026-08-16, out/2026-08-17
W03edge-resiliencew03-jwt-exp.tsout/2026-08-16, out/2026-08-17
W04edge-resiliencew04-edge-cache-stale.tsout/2026-08-16, out/2026-08-17
W05edge-resiliencew05-standby-replication.tsout/2026-08-16, out/2026-08-17
W06edge-resiliencew06-cold-dr-timing.tsout/2026-08-16, out/2026-08-17
W07edge-resiliencew07-breakglass-mint.tsout/2026-08-16, out/2026-08-17
W08edge-resiliencew08-refresh-race.tsout/2026-08-16, out/2026-08-17
W09edge-resiliencew09-auth-replication.tsout/2026-08-16, out/2026-08-17
W12edge-resiliencew12-realtime-probe.tsout/2026-08-16, out/2026-08-17
W13edge-resiliencew13-function-timeout.tsout/2026-08-16, out/2026-08-17
W14edge-resilienceRUNLOG-
W15edge-resiliencew15-ddl-during-replication.tsout/2026-08-16, out/2026-08-17
W16edge-resiliencew16-sequence-resync.tsout/2026-08-16, out/2026-08-17
W17edge-resiliencew17-config-parity.tsout/2026-08-16, out/2026-08-17
W18edge-resiliencew18-function-coldstart.tsout/2026-08-16, out/2026-08-17
W19edge-resiliencew19-render-failures.tsout/2026-08-16, out/2026-08-17
W20edge-resiliencew20-statement-timeout.tsout/2026-08-16, out/2026-08-17
W21edge-resiliencew21-spend-cap.tsout/2026-08-17
W22edge-resiliencew22-bulk-sync.tsout/2026-08-16, out/2026-08-17
W23edge-resiliencew23-cron-restart.tsout/2026-08-16, out/2026-08-17
W24edge-resiliencew24-edge-failover-proxy.tsout/2026-08-16, out/2026-08-17
W25edge-resiliencew25-routing-eject.tsout/2026-08-17
W26edge-resiliencew26-storage-dual-write.tsout/2026-08-17
RR02replica-routingrr02-replica-rerouting.tsrun-2026-10-23T07-57-32-539Z, run-2026-10-23T08-22-50-864Z
  1. PostgREST, “API Configuration and Custom Claims,” PostgREST Docs v11. https://docs.postgrest.org/en/v11/references/auth.html ↩ ↩2

  2. Supabase GitHub Discussion #48123, “Fresh Supabase Auth JWT rejected by PostgREST: PGRST303 ‘JWT issued at future’,” 2026-07-21. https://github.com/orgs/supabase/discussions/48123 ↩ ↩2

  3. Supabase, “Status - Incident History,” status.supabase.com. https://status.supabase.com/ ↩ ↩2 ↩3

  4. Supabase, “Connection management,” Supabase Docs. https://supabase.com/docs/guides/database/connection-management ↩

  5. Supabase status mirror, “Error spikes in one of the eu-east-1 Supavisor clusters,” 2026-08-24, Pulsetic. https://pulsetic.com/status/supabase/incidents/6635/ ↩

  6. supabase-lab, “Pooler-checkout experiment,” supabase-lab runlog, section “Where after 15000ms comes from” (2026-10-23). https://github.com/erfianugrah/supabase-lab/blob/d726c9a/experiments/pooler-checkout/RUNLOG.md ↩

  7. Supabase, “Billing FAQ - Fair Use Policy,” Supabase Docs. https://supabase.com/docs/guides/platform/billing-faq#fair-use-policy ↩

  8. Supabase, “Manage Storage Image Transformations usage,” Supabase Docs. https://supabase.com/docs/guides/platform/manage-your-usage/storage-image-transformations ↩ ↩2

  9. Supabase, “Navigating regional network blocks,” Supabase Blog. https://supabase.com/blog/navigating-regional-network-blocks ↩

  10. Supabase, “Automatic retries in supabase-js,” Supabase Docs. https://supabase.com/docs/guides/api/automatic-retries-in-supabase-js ↩ ↩2 ↩3 ↩4 ↩5

  11. Supabase, “Automatic PostgREST retries for transient errors,” Supabase Changelog. https://supabase.com/changelog/45071-automatic-postgrest-retries-for-transient-errors ↩ ↩2 ↩3

  12. Supabase, “Realtime Quotas,” Supabase Docs. https://supabase.com/docs/guides/realtime/limits ↩

  13. Supabase, “Service Level Agreement.” https://supabase.com/sla ↩ ↩2

  14. Supabase, “Read Replicas,” Supabase Docs. https://supabase.com/docs/guides/platform/read-replicas ↩