Supabase open-sources an eval suite for coding agents
Supabase released Evals, an Apache-2.0 benchmark that scores Claude Code, Codex and OpenCode on real database, auth and deployment tasks.
Supabase has published Evals, an open-source benchmark that measures how well coding agents handle real work on its platform, releasing both the harness and its first round of scores under Apache-2.0 in the supabase/evals repository.
How it works
The suite runs scenarios against containerised Supabase stacks rather than static prompts, and slices coverage three ways: by product (database, auth, storage, edge functions), by topic (row-level security, security posture, migrations, raw SQL), and by stage of work (build, deploy, investigate, resolve). Scoring mixes deterministic checks — did the migration apply, did the policy actually restrict access — with LLM-as-a-judge grading for the parts that resist assertion.
On build-stage tasks, Claude Code paired with Opus 5 and with Kimi K3 reached 100% without any Supabase-specific skills loaded. Smaller and mid-tier models depended much more on that scaffolding: Sonnet 5 went from 78% to 100% with skills enabled, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%. Supabase's summary of the pattern is that skills mattered least to the strongest models and most to the weakest.
The failure analysis is arguably more useful than the leaderboard. Agents consistently hand-wrote migrations instead of using declarative schema files, and verified authentication manually rather than calling the proper libraries — both habits that pass a test and fail in production. Documentation usage varied sharply too: Codex pulled roughly eight docs per scenario against about two for Claude Code.
Why it matters
Most agent benchmarks measure whether code runs. This one measures whether an agent operates a specific production platform the way its maintainers would, which is closer to what teams actually delegate. That makes it a template other infrastructure vendors can copy: a platform owner is uniquely positioned to define correct behaviour on its own surface, and publishing that as a runnable Apache-2.0 suite turns vendor documentation into an executable spec.
The skills result also lands on a live debate about where agent capability sits. If a well-written skill file closes a 22-point gap for a mid-tier model, then a meaningful share of what reads as model quality is really context engineering — and it is the cheaper half to fix.