PayBench: A Benchmark for Unsafe Commercial Autonomy

Toronto

Seeking Fundingactive1 person

Benchmark for agents spending human money - paybench.org

Project Details

What?
Paybench builds a benchmark for agentic payment safety. and a taxonomy of agentic safety failures.
Agents are run through 250 scenarios of trap-lookalike pairs of payment scenarios. these are scored against a human baseline of when it is safe to purchase autonomously, versus when not.

Who?
Conor Plunkett.

I built and sold an AI agent company for customer feedback to Crossmint in 2024. I work on agentic commerce infrastructure at Crossmint.
I quit last year, and am now pursuing AI research

Output?

A full benchmark scoring of frontier models against the 250 scenario set. Taxonomy of all agentic payment failues

Theory of Impact

An agent with $100,000 of capital to spend is far more dangerous to society than one with $100. But the same rules apply to each.
If we can provide a taxonomy of agentic payment failures at low $ amount, they should apply at a higher $ amount against rogue agents.
Toolkits, payment processes, spend rules apply regardless of $ amount. Paybench will figure out what tools and guardrails prevent agentic safety errrs.

People

Conor Plunkett

Team Member

Funding Details

Start Date: -
End Date: -
Expected Duration: 2 months
Funding Raised to Date: -
Annual Budget: -
Monthly Burn Rate: -
Current Runway: -
Funding Goal: -
Funding Stage: Seeking first grant
Fiscal Sponsor: -

Track Record

July 8th - human survey filled and scorecard created
july 1st - first successful runs of openai mini model on objective payment scenarios
june 25 - website live
june 20th - full experiment design posted

Grants Received– no grants recorded

Add grant

Discussion

Gavin LeechReviewer2mo

Hi Conor! Any plans to test for eval awareness?

Conor Plunkett2mo

Hi Gavin! Good question. No, not right now, and I probably should have thought of that.
I can definitely add that in. I'll look up the most recent methods now, thank you for it!