Comparisons / Test harnesses
RowShield vs pgTAP: hand-written RLS tests versus continuous verification
The short version
- pgTAP is a TAP-based unit-test framework running inside Postgres; teams write explicit assertions for policies and schema. RowShield is a scheduled monitor over the live catalog plus an anon-key probe, so verification happens without anyone maintaining a suite.
- Choose pgTAP when — you enjoy writing database tests, want exact control of edge cases, and have the discipline to keep the suite green as schema evolves.
- Choose RowShield when — your project changes weekly under generated code, nobody owns the test suite any more, and you want findings and remediation SQL instead of failing asserts at 2am.
RowShield rules relevant here
Head to head: pgTAP vs RowShield
| Capability | pgTAP | RowShield | Edge |
|---|---|---|---|
| What it verifies | Whatever your suite asserts — powerful when the author anticipated the case, silent about everything nobody thought to encode. | Nine shipped rule classes over every public table on every scan, plus live anon-key behaviour through the read-only probe. | RowShield |
| Runs against production | Typically a local or CI database rebuilt from fixtures, so it verifies your migration files rather than the project actually serving traffic. | Your actual catalog, read on schedule straight from pg_policies and pg_class, diffed scan over scan so drift cannot hide between releases. | RowShield |
| Maintenance burden | Every migration can break asserts, and suites rot fastest during busy weeks — precisely when nobody has spare hours to repair them. | Nothing for your team to maintain; rules are maintained upstream and apply to whatever shape your catalog takes next month. | RowShield |
| Failure signalling | Failing TAP output buried in job logs, visible only while someone still reads that pipeline at all. | Slack, Discord, email or webhook alerts, fired only when a finding is created, regresses or resolves — quiet otherwise. | RowShield |
| Expressiveness for exotic cases | Complete SQL freedom: triggers, roles, transaction isolation, anything Postgres can express — far beyond any fixed rule set. | Deliberately narrow: posture, behaviour and drift only; we are not a general test framework. | pgTAP |
| Setup cost | Install the extension, learn TAP conventions, then write and review the first hundred asserts. | Paste the project URL for the free probe; connect a read-only connection string for scheduled monitoring. Minutes, not sprints. | RowShield |
| Cost profile | Free and open source; the real price is engineer time spent writing and maintaining asserts over the years. | Free plan includes daily scans; paid plans add hourly cadence. | pgTAP |
Column claims about pgTAP are sourced below. Where the edge is theirs, the page says so — and the sections that follow explain why Supabase teams still pick RowShield.
What pgTAP does
pgTAP bundles assertion functions — is(), has_table(), has_index(), policy-aware helpers through extensions like supautils-style wrappers or plain SQL — executed by pg_prove or a DO block. It is respected infrastructure, used well beyond Supabase, and its precision is real: if you can express the invariant, pgTAP can assert it.
Mechanically a suite is plain SQL scripts: each opens a transaction, runs assertions, rolls back and emits TAP lines runners aggregate into pass counts. Adding a table means hand-writing asserts; nothing enumerates which tables remain unverified.
Its weakness is not capability but entropy. Suites written for week-one schemas assert against week-twelve reality, and the failures surface as red pipelines rather than security alerts.
Where the scopes differ
pgTAP verifies intent you encoded; RowShield verifies the deployment itself. The distinction matters because the failure mode we watch — a migration dropping a policy, a bucket flipping public, a fix quietly reverting — happens after the last time anyone ran the suite.
The sequence is concrete and ordinary: an agent-generated migration renames a column and rewrites a policy; the rename lands, the rewrite does not. The suite run against yesterday's fixtures passes while the deployed grants changed regardless.
Behaviour is the second gap: no pgTAP suite asks PostgREST, as the anon caller, whether rows actually flow. That question is our probe's entire job, GET-only, using access every visitor already has.
Why Supabase teams choose RowShield over pgTAP
Because the alternative to a maintained suite is not "no verification" — it is scheduled truth. Findings arrive classified with severity and generated fixes from your real columns, including FORCE ROW LEVEL SECURITY. Plans never withhold results; frequency is what scales from Free to Team.
And regressions get named. A problem that returns after being fixed alerts as regressed — process-level information no assert ever surfaces.
Onboarding follows the same shape everywhere: connect a read-only role, let the first scan establish the baseline, then work findings by severity instead of by assert backlog. Teams arriving from suite-heavy setups report the same turning point: asserts kept passing while posture moved underneath, until an independent scan named what the green pipeline had waved through.
Where pgTAP is the right choice
For complex trigger logic, custom functions, permission matrices spanning roles, or transactional edge cases, nothing replaces expressive hand-written tests — keep pgTAP for those. If the schema changes twice a year and an engineer genuinely enjoys the harness, it costs nothing and works.
The honest caveat is arithmetic: a suite covers the repository at commit time, while RowShield covers the interval between deploys — where posture incidents live. Most teams end up running both and stop treating a green suite as evidence about production.
Using both
They compose cleanly: pgTAP for invariants only engineers can state, RowShield for the posture baseline nobody has time to restate weekly. When both disagree about a table, investigate before dismissing either — one sees fixtures, the other sees production.
Frequently asked
- Is RowShield affiliated with the pgTAP project?
- No. RowShield is built by Veristria and is unaffiliated with pgTAP or its maintainers. pgTAP is referenced descriptively throughout this page because it is the framework many Postgres teams reach for first, and it remains the property of its project. Nothing on this page implies endorsement, partnership or shared code.
- Is RowShield a good pgTAP alternative?
- For continuous RLS and exposure monitoring, yes — that job needs no test authoring at all, and a monitor never rots. For expressive database unit-testing, pgTAP remains excellent and the two combine well: asserts for invariants engineers must state precisely, scheduled scans for the deployed baseline everyone else forgets to restate.
- Can RowShield replace our existing suite?
- It can absorb the posture assertions most suites carry — RLS state, policy sanity, exposure checks — leaving exotic logic tests in place if you want them. Teams typically migrate in stages: connect the project, let a week of scans establish the baseline, then retire the specs that duplicate findings. The free probe previews exactly what the monitor would flag before you commit.
Check your project in about ten seconds
Paste a URL. No signup, no writes, nothing stored.
Run the free audit