SupabaseOperations and accessreference
Supabase incidents: what a client can do - a resilience reference
The platform fails in a small number of recurring ways. Each class below has a mechanism, a client-visible signature, and a workaround tier - and each claim is marked as measured in the lab (with the number) or documented-only. The workarounds are implemented in the resilience runbook; the RPO/RTO/monthly-cost axis for the durability tiers is in the DR tiers reference.
Measured numbers come from the
supabase-lab pvlab harness -
experiments/edge-resilience (2026-08-15/16, W21/W25/W26 2026-08-17) and
experiments/platform-downtime (2026-08-04, n=1 per operation, one Micro
project in ap-southeast-1); the cutover drill ran cross-region
ap-southeast-2 -> ap-southeast-1.
Classes 2, 3, 4, 7 and 8 also carry measurements dated 2026-10-10 from six
further experiments, each with a RUNLOG (the published out/ artifacts of
three of them are linked in the Modules table):
client-retries (supabase-js 2.112.3 and 13 other versions on Bun 1.3.14,
faults injected by a local proxy, n=1 per case unless stated),
pooler-checkout (Micro, ap-southeast-1 and us-east-1, Pro org),
lifecycle-ops (one Micro project, ap-southeast-1, n=5 restarts),
mgmt-api-faults (supabase CLI 2.120.0, OpenTofu provider, the lab’s own
client, every fault injected, no real 5xx seen), hostname-path (one Pro
project, ap-southeast-1, one APAC vantage) and edge-runtime-auth (the
functions.invoke retry cell only). The Modules table maps each id to its test.
TL;DR: Durability has its own tier ladder, priced out in
the DR tiers reference: daily backups
(included, up to ~24h RPO), PITR (~2-minute worst-case RPO,
~$100-$400/month), warm standby (34ms-1s). Reads are cheap to
protect (edge cache, measured: byte-identical through a forced origin
outage). Writes and auth need a warm standby, which
works managed-to-managed with logical replication and sessions that survive
cutover via third-party auth (TPA, Supabase’s mechanism for trusting an external issuer’s tokens) registration (measured: 34ms-1s replication
lag, TPA resolution ~60-122ms). Platform-managed
schemas (auth.*, storage.*) do not replicate at any tested size, so
standby auth posture is token portability plus forced re-login, not user
replication. The JWT clock-skew incident class has no runtime workaround -
only exposure reduction (a token TTL lever that takes effect in ~6.5s) and
detection (the PGRST code, not the HTTP status). The built-in supabase-js
retries cover GET, HEAD and rpc({ get: true }) on 503, 520 and connection
resets only, with no default timeout (Class 8); a full transaction pool
fails the first statement after 60 s while the connect succeeds (Class 3).
Which do I pick
Section titled “Which do I pick”| You need to survive… | Tier | Cost | Evidence |
|---|---|---|---|
| Origin 5xx on read paths | Edge cache Worker | One Worker, no infra change | Measured, W04 |
| Transient 503 or 520 on a GET | Built-in supabase-js retries (2.102.0 and later) | On by default; a deadline is yours to add | Measured, CR01-CR04 |
Transaction pool full (ECHECKOUTTIMEOUT) | Session pooler on 5432 as fallback | Client-side switch | Measured, PC01-PC02 |
| JWT claim rejections (PGRST303) | TTL raise + code-aware alerting | Config change | Measured, W01/W03 |
| Auth outage with active sessions | Longer TTL (sessions keep validating) | Config change | Measured, W03 |
| The whole PostgREST/Kong API layer (PGRST303, routing rollouts) | Own auth + claims over the wire | Rewrites the client; loses SDK surface | Measured, rls-wire-claims C01-C03 (reference) |
| Full regional/project loss | Warm standby + cutover | Second project + rehearsal | Measured, W05 |
| Anything else | Cold DR (pg_dump) | Cron + object storage | Measured, W06 |
| Data loss after the fact | PITR (point-in-time restore) | ~$100+/mo + Small compute | Documented (tiers) |
| Bad DELETE / bad migration | PITR - the standby replicates the mistake in 34ms | ~$100+/mo | Documented |
| Org-wide 402 restriction | Fix billing (card / cap / usage) | - | Documented |
| Fleet-wide incident | Contract (SLA) | Enterprise plan | Documented |
Which classes apply to you
Section titled “Which classes apply to you”Most readers need six to nine of the thirteen classes, depending on their shape.
| Your shape | Reads | Writes / auth | Durability | Classes that apply |
|---|---|---|---|---|
| Free, one project | edge cache (7) | accept the outage | scheduled db dump - Free has no backups | 1, 2, 4, 5, 7, 12 |
| Pro, one project, production | edge cache (7) | break-glass mint (W07) + TPA prep | daily backups (7d) or PITR | 1-8, 12 |
| Team / Enterprise, production | edge cache + failover (7) | warm standby (9) | PITR + standby (tiers) | all |
| Enterprise, federated IdP | same | standby; fresh logins work via the shared issuer | PITR + standby | all; the auth.* caveat drops out |
Classes 3, 8, 10 and 11 are USAGE-gated, not plan-gated: they apply on any plan the moment you use the pooler, supabase-js, Edge Functions, or a query long enough to hit a timeout.
Class 1: JWT claim rejection (PGRST303)
Section titled “Class 1: JWT claim rejection (PGRST303)”PostgREST validates iat/exp on every request with a
30-second clock-skew tolerance.1 If the issuer’s clock runs
ahead of the validator’s, freshly minted tokens carry a future iat and
are rejected - while tokens minted before the skew began keep working. Auth
(/auth/v1/user) can return 200 while the Data API rejects the same token;
the two validate on different hosts.2
The client sees 401 {"code":"PGRST303","message":"JWT issued at future"},
and the failure reaches every active session as its access token expires and
refreshes - with the default 1-hour TTL, the whole user base goes through the
broken path within one TTL.
Measured in edge-resilience W01 (2026-08-15), the documented tolerance is exact: +30s accepted, +31s rejected (401 PGRST303). Expired tokens reject with the same code; an unknown signing key rejects PGRST301. Two concurrent refreshes of the same token both return 200 (W08) - simultaneous reuse does not hard-fail.
The Supabase status page logged “401 errors due to JWT rejections” on 2026-08-14 (“newly refreshed JWTs being rejected by the API”), with an earlier “elevated JWT authorization errors” incident the same week and a matching community report three weeks prior.32 Three occurrences in a month.
- Raise
jwt_expvia the Management API to reduce exposure. Readback is immediate for the new value; the issuer honours it ~6.5s later (measured, W03). The apply is async, so poll the readback after anyconfig/authPATCH including the restore - on 2026-08-17 an immediate readback returned the pre-restore value (W17). A 12h TTL means a 20-minute skew window touches a small fraction of sessions instead of all of them; the 12h figure is a choice, not a measurement. - Alert on the PGRST303 rate. The HTTP status (401) is ambiguous; the code identifies the class.
- Do not retry mid-skew; every refreshed token is equally future-iat. The default client makes exactly 1 attempt on a claim rejection (measured, W02, supabase-js 2.112.3).
- No runtime fix exists; the repair is platform-side (PostgREST update plus NTP hardening). The only structural alternative is leaving the validating layer entirely: claims over the wire keeps per-user RLS with your own verifier as the only clock (measured, rls-wire-claims C01/C02) - a rewrite, not a mitigation.
Class 2: Compute lifecycle downtime
Section titled “Class 2: Compute lifecycle downtime”On restart (platform-downtime D01/D03/D04, 2026-08-04, n=1 per operation,
one Micro project in ap-southeast-1, 500ms sampling), the REST and Realtime
probes had zero failed samples, Auth 75 s (HTTP 521), Storage 78 s
(HTTP 500), pooler 158 s (Failed to connect to database: {:error, :timeout}). The lab’s summary file (RELIABILITY.md) carried REST ~10s,
Storage ~26s, Auth ~75s as N=3 p50 dated 2026-08-06..08; the RUNLOG is
the record, and no artifact exists for those figures. A resize costs more
than a restart, by a different factor per path: Auth 131 s against 75 s (1.75x),
pooler 207 s against 158 s (1.31x).
Those REST and Realtime zeros were withdrawn on 2026-10-23. The REST probe
was an anon GET /rest/v1/, which the API gateway answers with 401 before
PostgREST or the database is reached, and the Realtime probe a WebSocket
handshake, so both said only that the HTTP tier answered. Re-measured with a
row read and a Postgres Changes join on throwaway Micro and Medium projects
(platform-downtime RUNLOG,
section 2026-10-23), REST reads and Realtime subscriptions failed for about
as long as Auth and Storage: restart 247 and 194 s, Micro to Small 171 and
218 s, Medium to Large 113 s for the read (a second run is void: the test
machine slept inside it, lab #211). Only a network restriction
left them alone. The per-path matrix is in what a platform operation
costs.
Re-check on 2026-10-10 (lifecycle-ops LO05, one Micro project in ap-southeast-1, five sequential restarts, each after a readiness check, same 0.5 s sampling; a window runs from the first failed sample to the first sample of a 5 s sustained recovery). The D01 row is the n=1 figure above, kept as it was; the two are different days and different projects, so the table is not a controlled comparison.
| Path | D01, 2026-08-04, n=1 | LO05, 2026-10-10, n=5: window p50 / max | LO05 per run (s, runs 1 to 5) |
|---|---|---|---|
| REST (HTTP tier) | 0 failed samples | 0 of 5 runs failed | - |
| Auth | 75 s | 54 s / 60 s | 32, 31, 54, 59, 60 |
| Storage | 78 s | 58 s / 59 s | 28, 29, 58, 59, 59 |
| Pooler (6543) | 158 s | 54 s / 151 s | 31, 151, 52, 54, 59 |
| Direct (5432) | not sampled | 50 s / 56 s | 28, 28, 50, 55, 56 |
The first failed sample came 2 to 3 s after the POST /restart on every
failing path. The control-plane status read ACTIVE_HEALTHY again at 40, 41,
61, 61 and 71 s (10 s poll), so it ended before the pooler recovered in run 2
(41 s against a 151 s pooler window). Windows were about 30 s in runs 1 and 2
(the pooler’s 151 s in run 2 excepted) and 50 to 60 s in runs 3 to 5; five
runs on one project cannot say whether that is drift across back-to-back
restarts, platform load or chance. Realtime was not sampled in LO05, and D01’s
158 s sits near LO05’s single 151 s pooler run, not near its other four.
LO05 re-ran on 2026-10-23 with a row read and a Postgres Changes join
(artifact):
REST reads and Realtime joins failed in 5 of 5 restarts, p50 150 s and 146 s,
max 181 s and 179 s, alongside Auth (p50 149 s), Storage (151 s), the pooler
(147 s) and direct (145 s), while the HTTP-tier probes failed in 0 of 5. That
day’s p50 windows (145 to 151 s) ran more than twice the 2026-10-10 p50s (50
to 58 s), though runs 3 and 4 (43 to 58 s) matched them; not explained.
POST /restart answered HTTP 200 each time; a second POST /restart sent
while the status read RESTARTING also answered 200 with an empty body (LO05c,
whether it queued was not sampled). A restart sent about 70 s after the
creating POST, to a project that already read ACTIVE_HEALTHY, was refused
with HTTP 400 Project restarts are only allowed ten minutes after the creation process has completed. followed by text the artifact cut off; a restart 300 s after the first ACTIVE_HEALTHY read on
another project was accepted (LO05f), so the boundary is unresolved. Probes on
the macOS test machine used dns.resolve6 for the AAAA-only direct host,
because dns.lookup (getaddrinfo) returned ENOTFOUND for it while a TCP
connect to the literal address worked.
For probe design, anonymous network probes see nothing (network accept is not readiness), and the Auth probe must be the signUp REST surface - RLS blocks the password grant on a fresh project, so a password-grant sensor reports an outage that is really its own permission model. pg_cron jobs resume across a project restart and keep firing on schedule - a 1/min heartbeat job had 2 rows before the restart and 5 after (3 new rows across the window), with no catch-up doubling (measured, edge-resilience W23, 2026-08-16). The other side of no catch-up: an instant that falls inside the restart is skipped, and a run in flight is recorded failed and not re-run (cron-behaviour CB04/CB05, 2026-10-28; see pg_cron behaviour).
Class 3: Pooler failure modes
Section titled “Class 3: Pooler failure modes”Measured in pooler-semantics and http-tier-lockdown, a session-mode startup auth failure is a clean 58P01 at connect. In transaction mode, connect succeeds, the first statement fails 08P01/0A000, and prepared statements are unsupported. The same HTTP 400 can be a bad request or a capacity ceiling - read the SQLSTATE. Non-pooler 5432 can accept and then silently blackhole; always set connect and statement timeouts.
Pooler checkout timeout
Section titled “Pooler checkout timeout”Measured 2026-10-10 in pooler-checkout (PC01, PC02; one Micro project per run,
ap-southeast-1, Pro org, Supavisor transaction pooler on 6543, one IPv4-only
vantage in Singapore). The Management API reports default_pool_size null for
the project, so the 16 below is a measured concurrency; no configured value was read
back.4
| Observation | Value | Module |
|---|---|---|
Active pg_sleep backends 6 s into 40 concurrent pg_sleep(25) clients | 16 in both runs | PC01b |
Client 17 (16 backends held by pg_sleep(75)): connect() | succeeds in 43 to 52 ms | PC01c |
Same client, first select 1 | fails after 60,007 to 60,009 ms (6 of 6 trials) | PC01c |
| Error text | XX000 (ECHECKOUTTIMEOUT) unable to check out connection from the pool after 60000ms in Transaction mode | PC01c |
Holders sleeping 8 s, client 17 sends select 1 2 s in | served after 6,018 to 6,056 ms (4 of 4), no error | PC01d |
select 1 after the holders ended | 7 to 8 ms | PC01c |
A connect probe or a ping on connect does not see exhaustion; the client is
accepted and the error arrives on the first statement. A statement that queues
for less than 60 s is served. A public status-page mirror quotes the same error
with after 15000ms for an incident on 2026-08-24 in one cluster;5
this project answered 60,000 ms in 6 of 6 trials. A 2026-10-23 re-run on a
Micro and a Small project (PC01, PC05-PC07) separated the two figures: the
after 60000ms pool-slot wait held at 60,006 to 60,013 ms whatever the client
set (node-postgres statement_timeout, connectionTimeoutMillis, a pool size
of 5), and after 15000ms came back in transaction and session mode when
Postgres logins were delayed 20 s, at 15,046 to 15,180 ms, with the pool not
full. The Supavisor source at commit d875a44f8316 (read 2026-10-22; the release
serving the projects was not read) puts a fixed 15,000 ms on handing a slot’s
backend connection to the client; that the public reports
were this path, and not a tenant configured with a 15 s pool wait, is
inference6. Role postgres cancelled a pg_sleep(130) with
canceling statement due to statement timeout at 120 s in an exploratory run
(not an artifact row), so holders must sleep under 120 s.
Fallback, with the 16 transaction-mode backends held (switch time is from the primary’s failure to the fallback’s answer; three trials per row in each of two runs):
| Fallback | Switch time | Module |
|---|---|---|
| Session pooler, 5432, after a 3,000 ms client deadline | 49 to 70 ms (total 3,092 to 3,131 ms) | PC02a |
| Session pooler, 5432, after the server’s own 60 s error | 56 to 117 ms (total 60,115 to 60,359 ms) | PC02c |
| Direct 5432, no IPv4 add-on | 0 of 3 answered in each run; ENOTFOUND in 6 of 6 trials | PC02b |
| Dedicated pooler (PgBouncer) 6543, IPv4 add-on on | 126 to 167 ms | PC02e |
| Direct 5432, IPv4 add-on on | 135 to 168 ms | PC02f |
The OS resolver on this vantage returned 0 IPv4 and 0 IPv6 addresses for the
direct host without the add-on, while a public resolver (1.1.1.1) returned 1
AAAA record (PC04d), so the ENOTFOUND is this vantage’s view of an IPv6-only
host. With the add-on enabled through PATCH billing/addons, this resolver
returned an IPv4 address after 80 s in both runs, the dedicated pooler answered
at 81 s and 80 s and direct at 81 s and 81 s (PC02d). One session-mode client
running pg_sleep(15) beside the 16 held transaction backends took a 17th
backend and completed (PC02h, n=1, pool setting unchanged), so one session-mode
client fit as a 17th backend; it was not compared with a documented combined
limit, and a herd of fallbacks was not run. Session mode behaves like a direct connection
for prepared statements and SET, and each fallback client holds a backend for
its whole connection.
Driver behaviour after a connection reset
Section titled “Driver behaviour after a connection reset”PC03 put toxiproxy between the client and the transaction pooler and ran 5
workers issuing select 1 every 200 ms for 22 s in a pool of 5, with a fault
2 s long (reset, cut) or 150 ms long (flap), 6 runs per cell. After the
fault cleared, 0 first attempts failed in the 111 runs that did not crash, for
node-postgres, postgres.js and Prisma: all three evicted dead connections and
recovered without a restart.
| Driver, fault | Rescued by retry-once at 0 ms | Rescued by retry-once at 300 ms |
|---|---|---|
node-postgres, flap (150 ms) | 0 of 30 | 25 of 25 |
Prisma, flap (150 ms) | 0 of 30 | 30 of 30 |
node-postgres, reset | 2 of 254 | 0 of 120 |
Prisma, reset | 2 of 263 | 30 of 120 |
postgres.js, reset | 30 of 30 | 30 of 30 |
A retry for a stale-connection failure has to wait longer than the outage it
follows: at 0 ms it landed inside the 150 ms the proxy was down (an inference
from the timing, not a separate probe). During a 2 s reset or cut, node-postgres
and Prisma failed 40 to 50 first attempts per run at 0 ms and 20 per run at
300 ms; postgres.js showed no failed first attempt in 9 of 12 cut runs, with
the slowest statement at 1,902 to 2,103 ms, about the length of the cut.
node-postgres Pool without an 'error' listener exited non-zero in 9 of 12
runs at the cut (Unhandled 'error' event ... Connection terminated unexpectedly); with the listener 0 of 36 runs crashed. Not measured: a real
Supavisor node replacement, prepared statements (postgres.js ran with
prepare: false), idle pools, TLS verification, other driver releases.
Pooler host prefix
Section titled “Pooler host prefix”On 2026-10-10 a fresh Pro project in each of two regions (ap-southeast-1 in
runs 1, 2 and 5, us-east-1 in run 7) reported an aws-0 pooler host, and the
same tenant on the aws-1 host answered XX000 (ENOTFOUND) tenant/user postgres.<ref> not found (535 to 648 ms, and 3,636 and 3,684 ms in
us-east-1) while the aws-0 host answered (PC04). The hypothesis that aws-0
hosts are the old ones is not supported by these four projects; older projects
and other regions were not measured. Read db_host from GET /v1/projects/{ref}/config/database/pooler rather than writing the host into
code. The lab’s lint flagged 2 of 2 aws-0 literals in its fixtures and
ignored dedicated and direct hosts.
Class 4: Capacity and control-plane events
Section titled “Class 4: Capacity and control-plane events”The 2026 public status history documents recurring multi-region project creation/resize/restart failures: April APAC (~1.3h), April us-east (~2h), June multi-region (“existing projects are not affected unless restarted or resized”), August us-east-2 project access.3 Existing projects usually keep serving.
Pre-provision a project pool if your product provisions per-tenant, and never couple app availability to control-plane availability. There is no runtime fallback.
Your own billing state belongs in this class. Under the Fair Use Policy, continually exceeding the Free quota, exceeding the Pro quota with the spend cap on, or overdue invoices restrict the whole org: projects paused, databases read-only, launches blocked, and every API request answered 402 with a restriction description in the body.7 Detection is the 402 code in the probe table; the spend cap is not a request-path breaker (W21: quota+5 renders all 200); the fix is a card, a cap toggle, or reduced usage. The full billing/cost picture is in the DR tiers reference.
Status page as a change gate
Section titled “Status page as a change gate”Measured in lifecycle-ops LO01 (2026-10-10, one fetch at 00:43 UTC, an earlier
fetch at 00:33 UTC saw the same shapes, no credential). The status page’s JSON
under /api/v2/ can gate a create, restart, resize or upgrade, with these
shapes:3
| What | Value |
|---|---|
Components in components.json | 189 |
| Region names present | 18, each occurring 10 or 11 times; no group, service or parent field |
| Non-region components | 5 (Auth, Billing & Payments, Dashboard, Logging & Observability, Realtime) |
| Component status | operational on all 189 |
Incidents in incidents.json | 25, oldest created 2026-08-24, 0 open; no component or region field |
incidents/unresolved.json | HTTP 404 |
summary.json | one scheduled maintenance, created 2026-06-29, still scheduled on 2026-10-10 |
A lookup of the component named for a region therefore returns one of 10 or 11
components with nothing saying which service it is, so a gate takes the worst
status over every match. All 189 read operational, so component status has no
positive case in the lab yet. Incident scope is in text only: the gate (12 unit
tests, fixtures) reads the title first and treats a region found only in an
update body as a warning unless the text also has lifecycle wording, because one
incident whose title named no region listed a dozen regions in a PostgREST
rollout message and the first version of the gate counted it as 94.6 hours of
refused restarts for ap-southeast-1. impact is ignored: replayed for
upgrade the gate would have refused 11 of the 25 incidents, 3 of them
published with impact none (including “Upgrade Issues”, whose update text says
the ability to upgrade projects was suspended), and for restart it refuses 10,
2 of them with impact none. The gate does not read maintenance windows, since
the one entry is 100 days old and still scheduled.
Replaying the 25-incident feed (1113 hours) through a restart gate, from incident text only because component status is not historical, would have refused ap-southeast-1 for 38.5 hours, eu-west-1 for 41.8 and us-east-1 for 205.1 (one incident, “Intermittent latency in Eastern US”, stayed open about 7 days); 38.5 hours was the minimum over the 18 regions. That classification is a heuristic and not ground truth. Not measured: component behaviour during an incident, how late the feed runs against an incident’s real start, and maintenance-window semantics.
Create, region fallback and abandoned requests
Section titled “Create, region fallback and abandoned requests”Measured in lifecycle-ops LO03 (three creates) and LO04 (one), ap-southeast-1.
Requests that must fail did so fast and created nothing: unknown region 400
(194 ms), unknown org slug 404 (80 ms), missing name 400 (93 ms), unknown
instance size 400 (55 ms). A fallback wrapper that fell through one 400 landed on
the second region; ACTIVE_HEALTHY came back at the first poll twice (2 s) and
at 145 s once (10 s poll, n=3, so both happen and the count says nothing on how
often). The project that read healthy at the first poll served REST, Auth,
Storage and the pooler, and direct 5 s later (LO05a). A provisioning or capacity
failure on the platform side cannot be induced from a client and was not run.
LO04 (n=1): a POST /v1/projects whose caller gave up after 1 s still created
the project, and an identical re-send 1 s later answered HTTP 400 Project with name "<name>" already exists in your organization. after 0.17 s; 60 s later
the organisation held exactly 1 project with that name, ACTIVE_HEALTHY. A
client that retries after a lost response gets a 400 it has to read as “probably
created, look it up by name”, and keeps the name stable across retries of one
logical create; a wrapper that adds a random suffix per attempt would not meet
that check (reasoned from the 400, not run). Not measured: case sensitivity, a
name freed by DELETE, two concurrent identical requests, an idempotency key.
Management API clients under injected faults
Section titled “Management API clients under injected faults”Measured in mgmt-api-faults (MF01-MF05, 2026-10-10, Pro org, ap-southeast-1):
a fault-injecting proxy in a local container sat between each tool and
api.supabase.com, with the CLI pointed at it through a profile file whose
api_url is the proxy, the OpenTofu provider through endpoint, and the
lab’s client through its base URL. Every fault is injected and no real 5xx
was seen, so how a tool reacts to a real 5xx body is not measured. Attempt
counts are proxy log rows; n=1 per cell unless stated.
| Client | GET | POST | PUT, PATCH, DELETE | Timeout |
|---|---|---|---|---|
Lab harness mgmt() (MF01) | 1 attempt on 500, 502, 503, 504; the status is returned | not run | not run | throws TimeoutError at its 30 s default (30001 ms) |
| supabase CLI 2.120.0 (MF03) | 6 attempts per call with no pause on 500 (also 502, 503, 504); 429 is 1 attempt | 1 attempt on 500, 502, 503, 504 and 429 | PUT and DELETE 6 attempts on 500; PATCH not reached by any command | 60 s per attempt; a GET held 100 s gave attempts at 0, 60, 120, 180, 240 and 300 s and exit 1 after 360220 ms |
OpenTofu 1.13.1 with supabase/supabase 1.11.0 (MF04) | 4 attempts over about 6 s on a persistent 503; 2 on one 500 or one 429 | 1 attempt on 500 | 1 attempt on 500 (PATCH, DELETE) | waited 130 s for a held create and exited 0 after 132978 ms; a GET held 100 s was 1 attempt, 103298 ms |
functionPresent() in the same harness retries 429 only, with a fixed 15 s sleep
(3 attempts in 30144 ms for two 429s), and a 500 on its read returns present: false with status 500, which a caller cannot tell from a missing function
without checking status (MF01c). A held POST is not resent by either tool: the
CLI exits 1 at 60 s (60448 ms for a 75 s hold) with the create already landed
(MF05a), and the provider waited out the hold (MF05b). Writes answered 502, 503,
504 or 429 in the provider, PATCH through the CLI and connection-level faults
were not run.
Failed creates leave orphans
Section titled “Failed creates leave orphans”A create answered 500 after upstream had created the project (injected by the
proxy forwarding the request and replacing the answer) leaves the client with an
error and the project upstream. In the provider (MF04b-c), apply exited 1 with 1
POST, state held 0 resources, and a plain re-apply answered 400 (same name in
the organisation) with the orphan still there; deleting the orphan through the
API let the next apply exit 0 in 3468 ms. The CLI’s projects create answered
500 after upstream created it did the same, 1 attempt, exit 1, 1 project with
the requested name (MF03). The same-name 400 is the only duplicate protection
observed, and a retry with a fresh name is not covered and was not run (MF02c).
A settings write that failed before upstream leaves the project in state without
its settings (MF04d0); one that landed but answered 500 converges at the next
plan (MF04d2); a DELETE that landed but answered 500 leaves a state entry that
destroys cleanly later, because a 404 on the GET and the DELETE is accepted
(MF04f2-f3). The harness create behaves the same way on an answer lost to its
own 4 s timeout (MF02a, 1 project afterwards).
Restore paths (PITR, Free-plan pause and unpause, and what the Management API exposes for each) are priced and compared in the DR tiers reference, which owns that tier ladder.
Class 5: Schema-cache wedge
Section titled “Class 5: Schema-cache wedge”Measured in http-tier-lockdown, a PostgREST schema-cache wedge returns
503 PGRST002 “schema cache load”; the fix is
pg_notify('pgrst', 'reload schema'). PGRST001 on an empty exposed schema
is not a wedge. ACTIVE_HEALTHY means the HTTP tier is up, not that
PostgREST can reach Postgres - probe a real table, never the status field.
Class 6: Storage render path failure modes
Section titled “Class 6: Storage render path failure modes”Measured in edge-resilience W19 (2026-08-16): all six outcomes, verbatim:
| Source object | Render URL (?width=32) | Plain public URL |
|---|---|---|
| Valid PNG | 200, resized | 200 |
corrupt.png (text bytes) | 400 InvalidRequest - “The source image is invalid or unsupported for rendering” | 200, original bytes |
| SVG | 200, source SVG unchanged (no rasterisation) | 200 |
The render path never produced a 5xx, and the original always serves, which the workaround relies on.
Pre-generate renditions at upload (the docs-recommended architecture; the billing fix doubles as a resilience fix against the render path). The per-project transformations toggle is a hard stop, though cached transformed images may still bill after disabling, so do not promise a clean zero.
The billing half is a commercial decision, not an incident class: transformations bill per distinct origin image per billing cycle ($5 per 1,000, 100 included on Pro and Team, count resets each cycle), a growing library re-bills every month it is viewed, and the cap trade-off - cap on means eventual restriction, cap off means an uncapped bill - is priced out in the DR tiers reference.8
Class 7: Edge caching as an outage absorber
Section titled “Class 7: Edge caching as an outage absorber”Measured in edge-resilience W04 (2026-08-15), a cache-first Cloudflare Worker served warm URLs 200 with byte-identical bodies while the origin was hard-down. Cold URLs failed.
Cloudflare Workers wraps TCP failures to unroutable addresses as a 403
response rather than a JS exception, so catch-based stale fallback never
fires for that failure mode; handle it in the status branch. The same wrap
affects failover logic: a >=500-only failover condition never trips on
an unroutable primary, because the failure arrives as a 403 response.
Measured in edge-resilience W24 (2026-08-16), the failover proxy ran end to end, every step HTTP 200: prime the probe URL (origin: primary) -> redeploy with the primary pointed at an unroutable address (origin: standby) -> restore the primary and probe immediately (origin: still standby - the HOLD_MS flap-damping window, persisted in the Cache API so it survives the redeploy) -> probe after the window (origin: primary). Three bugs, each measured, had to be fixed before the sequence ran green:
- Cache-first masks failover. The first iteration measured origin “none” on the prime and outage probes: the cache-first read ran before the failover logic, and HIT responses carry no origin information. A failover mode behind a cache-first read is masked by warm entries - skip the cache when failing over, and tag every response with the origin that served it.
- The 403 wrap defeats
>=500failover conditions (the wrap described above). The shipped worker fails over on 5xx, 403, or any non-ok in outage mode. - PostgREST 400s on unknown query params (treated as column
filters). The drill’s
?_w24=cache-buster, forwarded to the origin, 400’d every probe - and a 400 is not a failover condition, so the outage phase served 400s tagged primary. Strip probe-only params from the origin URL; keep them only in the cache key.
The hold window must exceed the redeploy-plus-settle path between the last outage probe (which refreshes the failure timestamp) and the holdover probe - ~11s observed, so HOLD_MS=15000 measured an expired window and 60000 is what the drill ships.
The multi-tenant routing variant of the drill worker (W25) isolates failures per tenant: a stale routing row pointed at a dead origin fails only that tenant (502 for it, 200 elsewhere), and the eject costs a redeploy (10.6s) while the table lives in an env var (measured, edge-resilience W25).
Custom hostname and a resolver that cannot resolve supabase.co
Section titled “Custom hostname and a resolver that cannot resolve supabase.co”A hostname you own in front of the project is the client-side lever the platform itself used when it published a proxy domain during a regional block, which restored PostgREST, Realtime and Edge Functions while auth callbacks and storage object URLs stayed affected through hardcoded domain references.9 Measured in hostname-path (2026-10-10, one Pro project in ap-southeast-1, one APAC vantage, n=1 per row; the custom-domain add-on behaviour itself is in Static hosting on Supabase):
- HP06 resolution: the project host is 2 A records, no AAAA, TTL 300, the same
set through the system resolver, 1.1.1.1, 8.8.8.8, 9.9.9.9 and DoH on
Cloudflare and Google. A custom hostname set as a CNAME (TTL 60) to
<ref>.supabase.coreturned the same 2 addresses on all six paths.supabase.cois delegated root, thenco.(4 NS), then 2 NS on Cloudflare, with no DS at.coand no DNSKEY, and an unused label under it answers NXDOMAIN (no wildcard). - HP07 rig: a recursive Unbound in Docker with a zone forward sending
supabase.co.to a dnsmasq that answers NXDOMAIN, and an app container whose only resolver is that Unbound. The app is supabase-js doing 6 operations (a DNS lookup, password sign-in, a REST read, a public Storage object, an Edge Function call and a Realtime subscription). Control, recursion intact: 6 of 6 ok on the custom host and on the project host. - With
supabase.coNXDOMAIN and the custom host a CNAME to the project host, the app on the custom host did 0 of 6 operations (getaddrinfo ENOTFOUND), and so did the app on the project host. The CNAME is in the answer and its target is NXDOMAIN, so a custom hostname that is a CNAME to<ref>.supabase.codoes not survive a resolver that cannot resolvesupabase.co. - With the CNAME replaced by A records holding the project host’s 2 addresses,
the same hostname answered
NOERRORwith 2 A records and the app did 6 of 6 operations. Before that, curl pinned to each address with SNI set to the custom host returned200on both. The platform status still read5_services_reconfigureda few seconds after the CNAME was removed. How long the addresses stay valid (they are shared anycast addresses), areverifyor certificate renewal without the CNAME, and whether the platform supports address records were not tested; this records a measured behaviour and recommends nothing. - A resolver that had the chain cached before the outage kept answering at every 30 s sample to 271 s, and failed first at 301 s, matching the 300 s TTL on the cached target A record (one resolver configuration, no serve-stale and no prefetch; a stale-serving resolver would differ and was not run).
Not simulated: an outage seen by some resolvers and not others, a failure at the
.co TLD (the custom hostname sits under a different TLD, so only the CNAME
target is affected), and ISP-side filtering. The auth side of the same hostname
(the callback moving to the custom host about 6 s after activation, and an access
token iss that stays on the project host) is measured in HP03 and HP04; the
auth mechanics are in Supabase Auth end to end.
Class 8: Client retry behaviour
Section titled “Class 8: Client retry behaviour”Measured in edge-resilience W02 against supabase-js 2.112.3: 401 PGRST303 ->
exactly 1 attempt (no retry amplification). 503 x3 then 200 -> 4 attempts,
success in ~7.0s. Connection refused -> surfaced in ~7.0s. The built-in
retries are on by default since v2.102.0; custom policies go through
fetch-retry.10 Which statuses and methods they cover was
measured on 2026-10-10 in client-retries (CR01-CR04): a local fault proxy sat
between the client and the project’s REST URL, so an injected status never
reached the platform and the proxy log, including X-Retry-Count, is the
source of every attempt count. supabase-js 2.112.3 on Bun 1.3.14 unless a row
names another version or Node v26.11.1; n=1 per case unless stated.
Statuses by method
Section titled “Statuses by method”| Answered by the server | GET, HEAD, rpc({ get: true }) | POST insert, PATCH, DELETE, upsert, default rpc() |
|---|---|---|
| 503 | 4 attempts at 1, 2, 4 s, X-Retry-Count 1, 2, 3 (7008 ms; CR01a2, CR01b3) | 1 attempt (CR01b1-b3) |
| 520 | 4 attempts, 7012 ms (CR01a3) | 1 attempt (CR01b1) |
| Connection reset | 4 client attempts, 7016 ms (CR01a5) | 1 attempt (CR01b1) |
| 525, 502, 504, 500, 429, 408, 521, 522, 524 | 1 attempt for each of the nine (CR01a4) | not run per status; the method is never retried |
| 409 | not in the measured set | not run |
401 PGRST303 | 1 attempt (W02) | not run |
A GET reset gave 4 client attempts but 5 requests on the wire on the shared
proxy in CR01a5 (the first request sent twice with no header change), and 4
wire requests in every version in CR03, where each case had a fresh mock server;
the cause of the extra request was not isolated. The 503 row holds with
Retry-After unset. With it, the header sets the spacing: Retry-After: 2 gave
a 2000 ms gap, 0 gave 0 ms, 40 gave 40000 ms (above the 30 s cap), and an
HTTP date was ignored with a 0 ms gap (CR01a6). The real Data-API-off 503
(PGRST002, with the schema list emptied through the postgrest config
endpoint of the Management API) carries Retry-After: 0 on every answer, so the
default client’s four attempts ran back to back (gaps of 1700, 1800 and 2000 ms,
each about the previous answer’s duration plus 40 ms, 7252 ms in total) rather
than at 1, 2, 4 s; a POST made 1 attempt and a GET with .retry(false) 1 (CR01e1-e3). The
gateway’s edge_logs showed 5 rows for those calls and no X-Retry-Count
(source = 'edge_logs'), so attempts are visible there only as rows (CR01f).
supabase.functions.invoke is a neighbour of this class and does not share the
policy: supabase-js 2.117.3 made 1 attempt and the function received 1 request
for an injected 503 (with and without Retry-After), 502, 500 and 429, 3 trials
per status, error class FunctionsHttpError (edge-runtime-auth ER02-retry-503).
A 503 produced by the platform’s own relay, network errors and other client
versions were not measured. Limits of the function side are in Edge Function
limits, one ceiling at a time.
Versions and opt-out names
Section titled “Versions and opt-out names”Per version, from a local mock with a fresh server per case (CR03, both runtimes), cells are attempts:
| supabase-js | GET 503 | GET 525 | POST 503 | .retry(false) | db: { retry: false } | db: { retryEnabled: false } |
|---|---|---|---|---|---|---|
| 2.101.1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 2.102.0 to 2.111.0 (10 versions) | 4 | 1 | 1 | 1 | 4 | 4 |
| 2.112.0, 2.112.3, 2.117.3 | 4 | 1 | 1 | 1 | 1 | 4 |
Retries begin at 2.102.0. db: { retry: false } is honoured from 2.112.0, so on
2.102.0 to 2.111.0 only .retry(false) on the query opts out. db.retryEnabled,
the name the changelog gives, changed nothing in any of the 14 versions. A
per-query .retry(true) beats db.retry = false (4 attempts, CR01c4).
Timeouts and deadlines
Section titled “Timeouts and deadlines”- There is no default timeout: a response held 10 s completed (10108 ms,
CR01d2), and a request the server accepted and never answered was still
pending when observation stopped at 330004 ms on Bun and 330016 ms on Node
(CR04, one trial each, default client, no
db.timeout, no signal). db.timeoutis per attempt, not per call: with 2000 and a 503 on every attempt the call made 4 attempts and took 7010 ms (CR01d4). An abort is not retried: a response held 3 s withdb.timeout = 2000ended at 2004 ms after 1 attempt (CR01d3).- Under Bun 1.3.14, the docs page’s own pattern of
.abortSignal(AbortSignal.timeout(n))created inline did not end a call that was sleeping between retries: a 5000 ms deadline with a 503 on every attempt ran 4 attempts to 7011 ms (CR01d5a), and the same with 2500 ms ran 4 attempts in about 7 s on every version from 2.102.0 (CR03). Node ended after 2 attempts at about 2.5 s. AnAbortControllerwithsetTimeout, or a timeout signal with anabortlistener attached at creation, ended at the deadline on both runtimes (CR01d5b and CR01d5c, 5005 ms and 5003 ms with 3 attempts). The cause is not isolated and no Bun fix or other Bun version was checked. A plainfetchwith an inlineAbortSignal.timeout(2000)against a held request aborted at 2002 ms (a scratch check, Bun and Node).
Policies the built-in retries lack
Section titled “Policies the built-in retries lack”CR02 ran two client policies on Bun. A deadline plus one hedged GET (built-in
retries off, one deadline signal with a listener, one extra request when the
first fails with status 0 or 5xx or has been outstanding 1500 ms) bounded a held
request to 1575 ms (first request held 10 s, deadline 5000 ms), and a request
never answered to the 4000 ms deadline (ended at 4003 ms, 2 requests, both
aborted). It sent 2 requests for a 525 where the built-in policy sends 1. Left
on under the hedge, the built-in retries sent 6 requests in 6 s against 2 with
them off (CR02f). The hedge is for GET only: a hedged insert into a table with
no unique key wrote 2 rows for one logical write, and a hedged primary-key
upsert wrote 1 (CR02g, the server made slow rather than the network). The
policy retries immediately; a pause or jitter was not tested, and neither was
retry load on the Data API connection pool, which the docs warn about.10
Refresh-then-retry on a 401 (one auth.refreshSession() and one repeat) recovered
an injected 401 PGRST303 with 1 refresh call and 2 REST requests (200 after
136 ms), and a real 401 from a corrupted access-token signature
(PGRST301) the same way; 401 on every request stopped after 1 refresh and
1 retry; a session revoked with POST /auth/v1/logout?scope=global returned
the original 401 after a failed refresh (refresh_token_not_found); five
parallel requests each answered 401 once made 1 refresh call (CR02h, one trial,
so client deduplication cannot be told from timing luck). The injected 401s are
not the platform’s: the incident class “fresh JWT rejected” was not reproduced.
Docs against runtime
Section titled “Docs against runtime”Neither public page lists 408, 409 or 504. This page’s earlier text said the
built-in set was 408/409/503/504; 408 and 504 each got 1 attempt on a GET
(CR01a4) and 409 was not measured. The docs page lists GET, HEAD and OPTIONS (OPTIONS
was not tested) with 503, 520 and network failures, and names the opt-out
db: { retry: false }; the changelog says GET and HEAD, up to 3 retries at 1, 2, 4 s
capped at 30 s, and names retryEnabled.1011
The spacing, the methods and the version match; retryEnabled does nothing and
db.retry needs 2.112.0 (above). Neither page, as read, mentions Retry-After. Filed
upstream: no.
Class 9: Warm standby and cutover (the HA tier)
Section titled “Class 9: Warm standby and cutover (the HA tier)”The auth findings in this class and in classes 1 and 8 are consolidated, with the refresh, third-party-issuer and self-hosted GoTrue measurements that postdate them, in Supabase Auth end to end.
The full HA path works on managed projects (edge-resilience W05, 2026-08-15, cross-region ap-southeast-2 -> ap-southeast-1):
- Logical replication works managed to managed, with a subscription on
the standby against the primary’s direct host (
db.<ref>.supabase.co). Initial sync ~3.1-6.5s on a small table; replication lag 34ms-1057ms across regions. The pooler cannot be the source - it fails at the tenant-identifier layer (ENOIDENTIFIER). - Sessions survive cutover without copying secrets once the primary’s OIDC issuer is registered as a TPA integration on the standby (resolves in ~60-122ms); primary-issued tokens then read the standby’s API. Copying the JWT secret is not an option anyway - the config API accepts the write and changes nothing (measured 2026-08-14).
- A first-time issuer’s key costs ~30s of PGRST301 before PostgREST trusts it; a previously-seen JWKS warms in ~300ms. Rehearse the cutover before you need it.
A standby replicates your mistakes too: a bad DELETE lands on the
standby in 34ms, so recovering from logical corruption needs PITR.
Cutover hygiene, all measured (2026-08-16 unless noted): sequences do
not replicate - the first post-cutover insert fails with a verbatim
duplicate-key error (duplicate key value violates unique constraint "w16_t_pkey"), and setval resync restores inserts (W16). DDL does
not replicate, and primary-side DDL stalls ALL table replication -
even rows not using the new column, because the apply worker holds
streaming changes for any table not yet in r state - until the same
DDL lands on the standby, which resumes replication in ~6.1s with
backfill and no subscription recreation; apply standby DDL first
(W15). Per-project auth config (SMTP, SITE_URL, redirect URLs, rate
limits, jwt_exp) does not follow a cutover - re-apply it via the
Management API; the diff is readable verbatim from
GET /v1/projects/{ref}/config/auth on both projects (W17). In Realtime,
dropping and recreating a table under an active subscription kills events for that table NAME
(new OID, stale channel metadata) - stable canary tables only (W12).
And the wedged-subscription recovery is ordered: disable ->
set (slot_name = none) -> drop subscription ->
pg_drop_replication_slot on the publisher - dropping without the
slot detach leaves the publisher slot pinning WAL.
W09 and W14 (2026-08-15/16) located the replication boundary. Sizing the
standby up ruled out worker exhaustion on micro: pg_settings DO scale with size (max_connections 60 -> 90,
shared_buffers 256MB -> 512MB) but max_worker_processes is 6 on both
micro and small - platform-fixed. The failure differs by
mode: with copy_data=true the initial sync stalls in
pg_subscription_rel state d forever, the sync worker hung at
IPC/BgworkerStartup with its publisher sync slot inactive; with
copy_data=false the WAL sender connects but received_lsn stays
NULL - zero changes stream either way, at any size. A custom non-public
schema (lab_schema.t) replicates in ~4s on the
same instances, while storage.buckets does not - and inserts via the
postgres role change nothing, so the writer role is not the filter.
The boundary is specific to platform-managed schemas (auth.*,
storage.*); size, worker count and schema privacy were each ruled out.
The standby auth options are TPA token portability for existing sessions,
user backfill through the admin API or direct SQL, or forced re-login for
fresh logins. POST /auth/v1/admin/users accepts the source’s bcrypt
password_hash and the user keeps their password (measured,
tenant-consolidation C03, 2026-08-04); copying hashes via direct SQL into
auth.users is the other path (not measured on a standby). The W09 run
(2026-08-15) posted its backfill users without a hash, so its “not
portable via the admin API” note was an assumption; C03 measured the
hash path.
A 1,000,000-row initial sync (W22) completed in 22.7s (12.5s on a later battery run), with streaming lag ~245-276ms immediately after, so the small-table lag figures still hold at that size, and the sync scales without the stall the earlier runs suggested.
The cold DR floor (W06) is pg_dump 12.4s and restore 6.4s for 10k rows through the pooler session host.
For break-glass access (W07), GET /v1/projects/{ref}/postgrest returns the
project’s jwt_secret, and a locally minted token authenticates against
the live API with zero Auth involvement. During an Auth outage that keeps
the API usable; the same read grants full token minting for the project.
Prefer TPA portability - and
rotate the secret after any break-glass use: the Management API read is
auditable, but the secret stays live until rotated.
Class 10: Edge function limits
Section titled “Class 10: Edge function limits”Measured in edge-resilience W13/W18 (2026-08-15/16), the wall clock is
a 150s idle timeout - a sleeper function holding the connection past it
dies with 504 IDLE_TIMEOUT (W13). Cold start costs ~1.4s on the first
invoke after deploy plus idle; subsequent idle-cold invokes land at
121-302ms. Over 5 cold
invokes at 60s gaps: p50 284ms, p99 1433ms; warm p50 over 20 invokes is
98ms (W18). Keep-warm pings recover the difference between 98ms and
~300ms and do not remove the 1.4s; long jobs belong on a queue plus cron
rather than in a request path. The deploy-time ceilings (functions per project, bundle
size by deploy path, the four secrets limits) and the failures that
only look like limits (silent loss under parallel deploys, the 413 that
both size ceilings return) are measured separately in Edge Function
limits, one ceiling at a time.
Class 11: Statement and lock timeout signatures
Section titled “Class 11: Statement and lock timeout signatures”Measured in edge-resilience W20 (2026-08-16), both surfaces return HTTP 400, and the SQLSTATE in the body is what tells them apart:
statement_timeout(2s setting againstpg_sleep(5)): verbatimERROR 57014: canceling statement due to statement timeout, 3467ms wall.lock_timeout(3s setting on an advisory-lock contender, two sessions): verbatimERROR 55P03: canceling statement due to lock timeout, 4533ms wall.
Alert on the codes; a 400 alone is ambiguous with ordinary bad queries.
A lock timeout raised by a schema change rather than by a query you wrote has its
own shape, and the lock you asked for decides whether it happens at all: a
validating ADD CONSTRAINT on a table that references auth.users holds
SHARE ROW EXCLUSIVE on auth.users and blocks sign-in until it commits, while
NOT VALID plus VALIDATE holds only ROW SHARE during validation
(the locks, and the two-statement path).
Class 12: Realtime quota events
Section titled “Class 12: Realtime quota events”Realtime limits are per plan (documented, 2026): Free 200 concurrent
connections / 100 messages per second; Pro 500 / 500; Pro without the
spend cap and Team 10,000 / 2,500; Enterprise beyond
that.12 Exceeding the message rate disconnects clients
with a tenant_events error, and supabase-js reconnects automatically
once throughput drops below the plan limit. It looks like an
outage, with sockets dropping en masse; the disconnect reason identifies
it as a quota, and the fix is plan or usage.
Class 13: What you cannot work around
Section titled “Class 13: What you cannot work around”Fleet-wide platform incidents have no client-side answer - the remedy is
contractual (SLA, degradation prioritisation). The platform
uptime SLA is Enterprise-only -
99.9% per product per month, measured per-project, per-region or globally
depending on the product, GA features only, with service credits of
10-30% of the affected service’s monthly fees (capped at 20% of the
trailing twelve months’ fees) as the sole remedy.13 Support SLAs
exist for Team and Enterprise: Urgent response is 24 hours 24/7x365 on
Team and 1 hour 24/7x365 on Enterprise Standard and Priority Plus;
business-hours limits start at High (Team: 1 business day for High and
Normal, 2 business days for Low). The full severity table is on the SLA
page.13 Everyone below Team is on best-effort. Read replicas are
read-only (Data API GET, or SQL over the replica’s pooler), have no
documented promotion path, and Auth always goes to the Primary, so they
absorb read load only.14
Multi-region active-active writes risk split-brain: dual-write
returns 200/200 with ~107ms skew, but a partial failure (primary 200 /
standby 400) is not atomic (W26), so fail over rather than dual-write.
What to do about it
Section titled “What to do about it”One row per practice, with the number or error it addresses and the module it rests on. Module ids resolve in the edge-resilience RUNLOG, the platform-downtime RUNLOG, the instance-sizing RUNLOG and the compute-disk RUNLOG. The platform-downtime rows are n=1 per operation, so they rest on the ordering of the paths, not on the seconds. The 2026-10-10 rows resolve in the client-retries RUNLOG, the pooler-checkout RUNLOG, the lifecycle-ops RUNLOG, the mgmt-api-faults RUNLOG, the hostname-path RUNLOG and the edge-runtime-auth RUNLOG; they are n=1 per cell unless the row gives a count, and the control-plane rows are injected faults.
| Practice | Evidence | Module |
|---|---|---|
| Size a maintenance window off the longer of the restart and resize figures. | Auth 131 s and the pooler 207 s resizing up, against 75 s and 158 s for a restart (2026-08-04); on 2026-10-23 Auth was down 247 and 192 s restarting against 171 and 218 s resizing Micro to Small. | platform-downtime D01 vs D03, 2026-08-04 |
| Do not count on PostgREST or Realtime to ride out a planned restart or resize. | Row reads and Postgres Changes joins failed for about as long as Auth and Storage: restart 247 and 194 s, Micro to Small 171 and 218 s, Medium to Large 113 s for the read (a second run is void). The 2026-08-04 “zero failed samples” was an anon GET /rest/v1/ the gateway answers with 401, and a WebSocket handshake. | platform-downtime D01, D03, D04, 2026-10-23 |
Read (EADDRNOTALLOWED) from the pooler as a restriction; the other strings do not name the operation. | On 2026-08-04 {:error, :timeout} was a restart, {:error, :econnrefused} a resize up, terminating connection due to administrator command a resize down, (EADDRNOTALLOWED) address not in tenant allow_list a restriction just applied; on 2026-10-23 restarts and resizes up both answered terminating connection due to administrator command. | platform-downtime pooler-mode table |
| Allow-list your pooler clients’ egress addresses as well as your direct ones. | A network restriction is enforced by Supavisor against the client address, so 6543 is covered and the refusal lands 1 s after sampling starts. | platform-downtime D02, two runs |
Retry the first write after ACTIVE_HEALTHY, and the first Storage calls on a fresh project. | 5 of 5 fresh projects refused the first POST /auth/v1/admin/users with 500 unexpected_failure and accepted the next attempt one poll tick later. Fresh-project Storage answers TenantNotFound and then 429 SlowDown for the first minutes. A pre-provisioned project pool (Class 4) needs both. | placement reference, Verified row ‘Create -> healthy, and healthy is not writable’ (2026-08-04, n=5, bash run not in the lab repo); supabase-lab AGENTS.md provisioning note (2026-08-03, 2 of 2, 500 "Database error checking email"); W21 |
Pass a smart region group on POST /v1/projects when capacity matters more than the exact city. | region_selection: {"type": "smartGroup", "code": "apac"} was accepted on a Pro org with 201, landed ap-northeast-2, healthy in 135 s. | instance-sizing I02, 2026-08-17 |
| Retry Realtime joins with backoff, and never drop and recreate a table under an active subscription. | A join after 11 other modules had the socket closed before the event arrived. After a drop and recreate, events for that table name stop (new OID, stale channel metadata). | W12 |
| Keep the tenant routing table in KV or D1 rather than an env var. | The env-var eject cost a redeploy at 10.6 s. The KV/D1 eject time itself is not measured. | W25, 2026-08-17; the KV/D1 half is a design choice |
Diff the whole config/auth payload between primary and standby rather than a named field list. | On GET /v1/projects/{ref}/config/auth, custom_oauth_max_providers defaulted to 32767 on one project generation and 3 on another. | W17 baseline diff, 2026-08-17 |
Poll the readback after any config/auth PATCH, including the restore in a drill. | The apply is async and an immediate readback returned the pre-restore value. | W17, 2026-08-17 battery (24/25, green on re-probe) |
Check max_replication_slots before adding a subscription and drop orphan slots on the publisher first. | The limit is 10 on micro and on small. Each subscription pins one, and a dropped subscription can leave its slot on the publisher either way - after a plain drop (W05) and after slot_name = none then drop (W14) - so list pg_replication_slots on the publisher and pg_drop_replication_slot any orphan. | compute-disk D01; W05, W14 |
Add 504 and 408 to the GET retry set through fetch-retry; 520 is already retried. | The built-in GET, HEAD and rpc({ get: true }) retries cover 503, 520 and connection resets, 4 attempts at 1, 2, 4 s. 504, 408, 525, 502, 500, 429, 521, 522 and 524 each got 1 attempt (injected, supabase-js 2.112.3, 2026-10-10). This row’s earlier text, 408/409/503/504, was not what either docs page says1011; 409 was not measured. | CR01, 2026-10-10 |
Opt out with .retry(false); db: { retry: false } needs supabase-js 2.112.0. | .retry(false) gave 1 attempt from 2.102.0; db: { retry: false } was ignored through 2.111.0; db: { retryEnabled: false }, the changelog’s name, did nothing in 14 versions. | CR01, CR03 |
Bound every call with an AbortController and setTimeout, not an inline signal. | There is no default timeout (still pending at 330 s on Bun and Node, n=1 each) and db.timeout is per attempt (4 attempts, 7010 ms at 2000). On Bun 1.3.14 an inline AbortSignal.timeout did not end a call sleeping between retries (4 attempts, about 7 s for a 2.5 s or 5 s deadline); Node did. | CR01 (d4, d5), CR03, CR04 |
| Hedge only GETs, with the built-in retries off under the hedge. | A hedged GET bounded a held request to 1575 ms; with the built-in retries left on it sent 6 requests in 6 s against 2. A hedged plain insert wrote 2 rows, a hedged primary-key upsert 1. | CR02 (f, g), Bun only |
Write the retry for functions.invoke yourself; the client makes 1 attempt. | 1 client attempt and 1 function-side request on an injected 503, 502, 500 or 429, 3 trials per status, supabase-js 2.117.3. A platform-relay 503 was not measured. | ER02 (ER02-retry-503) |
| Send the first statement on a full transaction pool through 5432 session mode. | Client 17 connects in 43 to 52 ms and its first statement fails at 60,007 to 60,009 ms (6 of 6); the session pooler answered 49 to 70 ms after the primary’s failure and as a 17th backend. Direct 5432 is ENOTFOUND for an IPv4-only client without the add-on. | PC01, PC02, 2026-10-10 |
| Wait longer than the outage before the one retry of a stale pooled connection. | After a 150 ms flap a retry at 0 ms rescued 0 of 30 on node-postgres and on Prisma; at 300 ms it rescued 25 of 25 and 30 of 30. | PC03 |
Attach an 'error' listener to every node-postgres Pool. | A Pool without one exited non-zero in 9 of 12 runs at a 2 s cut; with one, 0 of 36 runs crashed. | PC03 |
Read db_host from the pooler config endpoint, not a literal aws-0 host. | Fresh Pro projects in two regions reported aws-0 and the aws-1 host answered ENOTFOUND tenant/user. Older projects and other regions were not measured. | PC04 |
| Look a project up by name before retrying a create, and keep the name stable. | An abandoned create still created the project and an identical re-send answered 400 already exists (n=1). A provider create answered 500 after upstream created it left 0 resources in state, an orphan, and a 400 on every re-apply until the orphan was deleted. A fresh-name retry was not run. | LO04, MF02, MF04 |
| Gate lifecycle calls on the worst status over every component named for the region. | 189 components carry 18 region names 10 or 11 times each with no group field, incidents.json has no region field, and impact was none on incidents that suspended upgrades. A replayed restart gate would have refused ap-southeast-1 for 38.5 hours of 1113 (heuristic). | LO01, 2026-10-10 |
Do not read control-plane ACTIVE_HEALTHY as the end of a restart outage. | Status was back at 40 to 71 s while the pooler ran to 151 s in one of 5 restarts; Auth 54 s and Storage 58 s at p50 (2026-10-10). In the 2026-10-23 re-run the status read healthy at 172, 142, 41, 41 and 141 s, each time before the REST read recovered (windows 181, 154, 46, 46 and 150 s from a first failure 1 to 3 s after the POST). | LO05; LO05 2026-10-23 artifact |
| Do not rely on a CNAME to the project host as the only path through a DNS outage. | With supabase.co NXDOMAIN at the resolver a CNAME custom hostname did 0 of 6 operations; A records holding the 2 project addresses did 6 of 6. A resolver with the chain cached failed first at 301 s (300 s TTL). Certificate renewal without the CNAME was not tested. | HP07, 2026-10-10 |
| Cap Management API calls yourself; the CLI waits 360 s on a stalled GET. | The CLI retries GET, PUT and DELETE 6 times with no pause on a 500 and POST once, 60 s per attempt (a held GET took 360220 ms); the provider retries GET only; the harness mgmt() never retries. Every fault was injected; no real 5xx was seen. | MF03, MF04 |
Not measured: the KV/D1 eject time (W25 measured the
env-var redeploy path only); whether a primary-side network restriction
blocks a standby’s walreceiver (D02 restricted the pooler, the direct-host
subscription path was never restricted); failing back by reversing the
replication direction; the 12h jwt_exp value (the lever is measured,
the value is a choice). Read replicas kept GETs answering, stale, while the
primary restarted, but not while the replica did: the restart took both nodes
down one after the other, and while the replica was down the load balancer
failed every read and sent none to the primary (replica-routing RR02i, 2026-10-23, two runs; see
read replicas). The discrepancy this
page used to carry between compute-disk D09 (0 s on /auth/v1/health across
four resizes, 2026-08-19) and platform-downtime D03 (131 s of HTTP 521,
2026-08-04) is resolved: the old D09 probe sent no apikey and dropped
outages still open at a phase end, and its 2026-10-23 re-run with the anon
key (250 ms sampling) saw /auth/v1/health down 152.375 to 254.869 s per resize, n=1 per
op (compute-disk RUNLOG). No module has probed
an authenticated Auth operation across a resize.
Decision guide
Section titled “Decision guide”Text version of the decision guide, keyed on what is failing:
- Reads 5xx or the origin down -> a cache-first Worker (W04).
- A 401 PGRST303 wave -> raise
jwt_exp(W03) and alert on the code. - Logins or refresh dead -> a longer TTL, where existing tokens still validate.
- Project or region lost -> a warm standby and TPA cutover (W05), rehearsed.
Evidence table
Section titled “Evidence table”| Claim | Value | How it was checked |
|---|---|---|
| Skew tolerance boundary | +30s pass, +31s PGRST303 | Measured (W01, two runs) |
| jwt_exp acceptance-to-effect | ~6.5s | Measured (W03) |
| supabase-js on PGRST303 | 1 attempt (supabase-js 2.112.3) | Measured (W02, mock; the other statuses are the CR rows below) |
Built-in retries, GET, HEAD, rpc({ get: true }) | 503, 520, reset: 4 attempts at 1, 2, 4 s; 525, 502, 504, 500, 429, 408, 521, 522, 524: 1 | Measured, injected statuses on a local proxy, supabase-js 2.112.3, Bun 1.3.14 (CR01, 2026-10-10, n=1 per case); docs: 503, 520, network failures, not 408, 409 or 5041011; filed upstream: no |
Built-in retries, POST, PATCH, DELETE, upsert, default rpc() | 1 attempt on 503, 520 and reset | Measured (CR01) |
Retry-After on a 503 | 2 -> 2000 ms gap, 0 -> 0, 40 -> 40000 ms (above the 30 s cap), HTTP date ignored; real Data-API-off 503 sends 0, 4 attempts in 7252 ms | Measured (CR01) |
| Opt-out names by version | .retry(false) from 2.102.0; db: { retry: false } from 2.112.0; db.retryEnabled none of 14 versions | Measured (CR03, Bun and Node) |
| Default timeout | none: pending at 330004 ms on Bun, 330016 ms on Node; db.timeout per attempt | Measured (CR04, one trial each; CR01) |
Inline AbortSignal.timeout between retries | Bun 1.3.14: 4 attempts, about 7 s for a 2.5 s or 5 s deadline; Node: 2 attempts at about 2.5 s | Measured (CR01, CR03) |
| Deadline plus one hedged GET | held request 1575 ms, hang 4003 ms at a 4000 ms deadline; built-in retries on under the hedge 6 requests in 6 s vs 2; hedged insert 2 rows, hedged PK upsert 1 | Measured (CR02, Bun only) |
| Refresh-then-retry on a 401 | 1 refresh call and 2 REST requests; 5 parallel 401s made 1 refresh | Measured (CR02; the parallel case is one trial) |
functions.invoke on 503, 502, 500, 429 | 1 client attempt, 1 function-side request, 3 trials per status | Measured (ER02, supabase-js 2.117.3, injected) |
| Edge cache under origin outage | 200, byte-identical | Measured (W04, forced outage) |
| Standby replication lag | 34ms-1057ms | Measured (W05, three runs) |
| Standby initial sync, 1M rows | 22.7s / 12.5s; lag after ~245-276ms | Measured (W22, two runs) |
| auth./storage. replication | zero changes at any tested size | Measured (W09/W14) |
| DDL on primary stalls all table replication | resume ~6.1s after standby DDL | Measured (W15) |
| Post-cutover insert without resync | duplicate key; setval fixes | Measured (W16) |
| Edge function cold start | ~1.4s first-after-idle only; p50 284ms cold vs 98ms warm | Measured (W18) |
| Edge function wall clock | 150s idle -> 504 IDLE_TIMEOUT | Measured (W13) |
| Render path on invalid source | 400 InvalidRequest; original still serves | Measured (W19) |
| statement_timeout signature | 57014 verbatim, 3467ms wall | Measured (W20) |
| lock_timeout signature | 55P03 verbatim, 4533ms wall | Measured (W20) |
| pg_cron across restart | resumes on schedule (2 -> 5 rows), no doubling | Measured (W23) |
| Edge failover sequence | primary -> standby -> holdover -> primary, all 200 | Measured (W24) |
| Tenant routing isolation | poisoned row: 502 for its tenant, 200 elsewhere; eject = redeploy (10.6s) via env-var table | Measured (W25) |
| Storage dual-write | parallel 200/200, ~107ms skew, bytes equal; partial failure not atomic; sync-after 97ms | Measured (W26) |
| Spend cap enforcement | none at the request path (quota+5 renders all 200) | Measured (W21) |
| TPA resolution on standby | ~60-122ms | Measured (W05) |
| Cold kid trust on cutover | ~30s PGRST301 | Measured (W05 full suite) |
| Cold DR, 10k rows | dump 12.4s, restore 6.4s | Measured (W06) |
| Break-glass minting works | 200 real / 401 wrong secret | Measured (W07) |
| Concurrent refresh race | both 200 | Measured (W08) |
| Restart gaps REST (HTTP tier)/Realtime (handshake)/Auth/Storage/pooler | 0 failed samples / 0 / 75 s / 78 s / 158 s | Measured (platform-downtime D01, 2026-08-04, n=1, 500 ms); RELIABILITY.md carried ~10s / ~26s / ~75s (N=3) with no artifact; re-checked 2026-10-10 in the next row |
| Restart re-check REST (HTTP tier)/Auth/Storage/pooler/direct (Micro, ap-southeast-1, n=5) | 0 failed runs / 54 s / 58 s / 54 s / 50 s window p50; max 60 / 59 / 151 / 56 s; first failure 2 to 3 s after the POST; control plane ACTIVE_HEALTHY at 40 to 71 s | Measured (LO05, 2026-10-10, 0.5 s sampling, Realtime not sampled) |
| Restart, data path: REST read/Realtime join (Micro, ap-southeast-1, n=5) | failed 5 of 5; window p50 150 s / 146 s, max 181 s / 179 s; HTTP-tier probes 0 of 5 | Measured (LO05 re-run, 2026-10-23, 0.5 s sampling) |
| Restart refused after a create | 400 about 70 s after the creating POST; accepted 300 s after the first ACTIVE_HEALTHY read on another project | Measured (LO05, two projects, boundary unresolved) |
| Resize up Auth/Storage/pooler | 131 s / 127 s / 207 s | Measured (platform-downtime D03, 2026-08-04, n=1) |
Fresh project: first write after ACTIVE_HEALTHY | 5 of 5 refused with 500, next attempt accepted | Measured elsewhere (placement reference, Verified row ‘Create -> healthy, and healthy is not writable’ (2026-08-04, n=5, bash run not in the lab repo); supabase-lab AGENTS.md provisioning note (2026-08-03, 2 of 2, 500 "Database error checking email"); summarised in operation cost) |
| Smart-group region selection on a Pro org | 201, landed ap-northeast-2, healthy in 135 s | Measured (instance-sizing I02, 2026-08-17) |
max_replication_slots on micro and small | 10 / 10 | Measured (compute-disk D01) |
| Pooler checkout with 16 backends held (Micro) | client 17 connects in 43 to 52 ms; first statement fails at 60,007 to 60,009 ms with ECHECKOUTTIMEOUT (6 of 6); a statement queued under 60 s is served | Measured (PC01, 2026-10-10; a public status mirror quotes 15000ms for other clusters, not reproduced) |
| Fallback to the session pooler on 5432 | 49 to 70 ms after the primary’s failure, 56 to 117 ms after the 60 s error; direct 5432 ENOTFOUND for IPv4-only without the add-on | Measured (PC02) |
| Driver pools after a TCP reset | 0 failed first attempts after the fault cleared (111 runs); retry-once at 0 ms 0 of 30 after a 150 ms flap, at 300 ms 25 of 25 and 30 of 30; Pool without an 'error' listener exited in 9 of 12 runs | Measured (PC03) |
| Pooler host prefix | API reported aws-0 on fresh projects in two regions; aws-1 answers ENOTFOUND tenant/user | Measured (PC04, four projects) |
| Status-page shape and change-gate replay | 189 components, 18 region names 10 or 11 times each, no group field; impact ignored; replay would refuse ap-southeast-1 for 38.5 h of 1113 | Measured (LO01, one fetch at 00:43 UTC; the replay is a heuristic) |
Create: bad bodies and ACTIVE_HEALTHY timing | 400, 404, 400, 400 in 55 to 194 ms, nothing created; healthy at 2 s twice and 145 s once | Measured (LO03, n=3) |
| Abandoned create re-sent | project created; identical re-send 400 already exists | Measured (LO04, n=1) |
| supabase CLI 2.120.0 retries | GET, PUT, DELETE 6 attempts on 500 with no pause; POST 1 on 500, 502, 503, 504, 429; 60 s per attempt | Measured (MF03, injected faults, Pro org, ap-southeast-1) |
| OpenTofu provider 1.11.0 retries | GET 4 attempts on a persistent 503, 2 on one 500 or 429; POST, PATCH, DELETE 1 on 500; a 130 s held create exited 0 | Measured (MF04, MF05, injected faults) |
Lab harness mgmt() | 1 attempt on 500, 502, 503, 504; TimeoutError at 30001 ms | Measured (MF01, injected faults) |
| Create answered 500 after upstream created it | provider state 0 resources, 1 orphan, re-apply 400 until deleted; CLI exit 1 with 1 project | Measured (MF02, MF03, MF04, injected) |
Custom hostname as a CNAME with supabase.co NXDOMAIN at the resolver | 0 of 6 operations (ENOTFOUND); A records holding the 2 project addresses 6 of 6; cached chain failed first at 301 s | Measured (HP07, HP06, one resolver rig, n=1) |
| 30s documented tolerance | documented | PostgREST v11 docs1 |
| Transform billing model | $5/1000 distinct origins/cycle | Supabase docs8 |
All measured rows come from the
supabase-lab pvlab harness
(experiments/edge-resilience),
2026-08-15/16/17, and are reproducible from the experiment’s Makefile -
the full run log with every number is
RUNLOG.md.
Batteries: 2026-08-16, 22 modules, 22/22
(out/2026-08-16);
2026-08-17, 25 modules, 24/25 with W17 green on re-probe after its readback
race was fixed
(out/2026-08-17).
The platform-downtime, instance-sizing and compute-disk rows cite their
RUNLOGs at the same commit, and none of the modules they cite here has a
published artifact: compute-disk has no out/ directory, and the
platform-downtime and instance-sizing out/ directories hold D05 and I04
only, neither of which this page cites.
The client-retries, pooler-checkout, lifecycle-ops, mgmt-api-faults,
hostname-path and edge-runtime-auth rows are dated 2026-10-10 and are quoted by
hand from their RUNLOGs. Their run artifacts are published redacted under each
experiment’s out/ directory and linked per module in the Modules table.
Modules
Section titled “Modules”References
Section titled “References”-
PostgREST, “API Configuration and Custom Claims,” PostgREST Docs v11. https://docs.postgrest.org/en/v11/references/auth.html ↩ ↩2
-
Supabase GitHub Discussion #48123, “Fresh Supabase Auth JWT rejected by PostgREST: PGRST303 ‘JWT issued at future’,” 2026-07-21. https://github.com/orgs/supabase/discussions/48123 ↩ ↩2
-
Supabase, “Status - Incident History,” status.supabase.com. https://status.supabase.com/ ↩ ↩2 ↩3
-
Supabase, “Connection management,” Supabase Docs. https://supabase.com/docs/guides/database/connection-management ↩
-
Supabase status mirror, “Error spikes in one of the eu-east-1 Supavisor clusters,” 2026-08-24, Pulsetic. https://pulsetic.com/status/supabase/incidents/6635/ ↩
-
supabase-lab, “Pooler-checkout experiment,” supabase-lab runlog, section “Where
after 15000mscomes from” (2026-10-23). https://github.com/erfianugrah/supabase-lab/blob/d726c9a/experiments/pooler-checkout/RUNLOG.md ↩ -
Supabase, “Billing FAQ - Fair Use Policy,” Supabase Docs. https://supabase.com/docs/guides/platform/billing-faq#fair-use-policy ↩
-
Supabase, “Manage Storage Image Transformations usage,” Supabase Docs. https://supabase.com/docs/guides/platform/manage-your-usage/storage-image-transformations ↩ ↩2
-
Supabase, “Navigating regional network blocks,” Supabase Blog. https://supabase.com/blog/navigating-regional-network-blocks ↩
-
Supabase, “Automatic retries in supabase-js,” Supabase Docs. https://supabase.com/docs/guides/api/automatic-retries-in-supabase-js ↩ ↩2 ↩3 ↩4 ↩5
-
Supabase, “Automatic PostgREST retries for transient errors,” Supabase Changelog. https://supabase.com/changelog/45071-automatic-postgrest-retries-for-transient-errors ↩ ↩2 ↩3
-
Supabase, “Realtime Quotas,” Supabase Docs. https://supabase.com/docs/guides/realtime/limits ↩
-
Supabase, “Service Level Agreement.” https://supabase.com/sla ↩ ↩2
-
Supabase, “Read Replicas,” Supabase Docs. https://supabase.com/docs/guides/platform/read-replicas ↩
Linked from
- A backend-for-frontend fan-out on Supabase Edge Functions
- Edge Function limits, one ceiling at a time
- Foreign keys to auth.users: the lock that freezes sign-in, and the NOT VALID path
- One envelope-encrypted app on managed and self-hosted Supabase
- pgjdbc and HikariCP on Supabase connection paths
- RLS without Supabase Auth: JWT claims over the wire
- Static site hosting on Supabase
- Supabase Auth end to end: tokens, trust, and what you can take in-house
- Supabase Auth providers: custom OIDC, passkeys, SAML and a sign-in canary
- Supabase CLI 2.120.0: pg-delta vs migra, config pull and the no-Docker stack
- Supabase data residency and sovereignty
- Supabase disaster recovery tiers: daily backups, PITR, warm standby - RPO, RTO, and cost
- Supabase read replicas: how requests reach them and what they cost
- Supabase resilience runbook: probes, TTL levers, edge cache, warm standby
- What a Supabase platform operation costs a client
- Where should a tenant live: one Supabase project per tenant, or many