Refundly: an AI agent that won't say 'done' until PayPal agrees

Carson Rodrigues

Carson Rodrigues / November 21, 2026

6 min read • ––– views

hackathonsai-agentspaypalfintech
The Refundly console: a 15.8% silent-failure rate, with 50.0% of tasks reported done and 44.7% verified

Asking a company for your money back is easy. Knowing the money really moved is the hard part.

Refundly is an AI agent that issues refunds and cancels subscriptions through PayPal, and it never says "done" until PayPal's own record agrees. I built it for the PayPal AI Hackathon, as a rework of an earlier project of mine called Attest Fleet.

CompetitionPayPal AI Hackathon, run on Devpost
OrganiserPayPal
WhenSubmission window October to November 2026. Judging runs 1 to 15 December, so final results aren't out yet
Built inOctober to early November 2026, as a rework of an earlier project
TeamSolo
Money spentNot tracked for the model calls, which ran through OpenRouter. PayPal ran in its sandbox, which uses test money. The one rented-GPU job, a small probe experiment, cost about $0.03
Codegithub.com/rodriguescarson/refundly, with a live console and a 2:55 demo video

The problem

Picture Rekha, 58. A gym app charged her twice and hid the cancel button. After twenty minutes on hold, she gave up. (Rekha is made up, but she's a very common person.)

An AI agent can do that chore. The danger is that it does it badly and says it did it well. A tool returns "success", the money never moves, the agent reports "refund issued", and its scoreboard stays green. I call that a silent failure: the agent says done, and the real record says not done.

What I built

A plain-language request goes through a small team of agents. An agent here just means a program where an AI model decides the next step and calls tools.

The controller reads the request and splits it into tasks. It can only read, so it can't touch money. Two workers each get one job. The billing worker can issue refunds and nothing else. The account worker can cancel subscriptions and can't refund. If someone tricks one worker with a sneaky message, it has no tool to do the other worker's job.

Before any money moves, a fixed rule check runs. Refunds above a limit (a default of $100) wait for a person to approve. A kill switch stops every write. Irreversible actions always go to a human.

Then the part I care about most. After the worker acts, a verifier asks PayPal directly. A refund only counts when PayPal's record of it says completed. A cancellation only counts when PayPal says the subscription is cancelled. No AI model decides that verdict. It's a plain comparison, so anyone who doubts a result can read the same record.

Flow diagram: customer asks, controller plans, worker agent acts, rule check, PayPal moves the money, verifier re-reads PayPal and says done only if PayPal agrees

The models are Gemini 2.5 Flash for the controller and workers, and Claude Haiku 4.5 as an auditor from a different model family, so the checker doesn't share the workers' blind spots.

Most of this came from the earlier project. What I rebuilt for the hackathon was the real PayPal connection, pointing the verifier at PayPal instead of a store I wrote myself, swapping the whole AI backend, and a new product aimed at ordinary customers.

Research

Primary research. I ran a 40-ticket test sweep, then one live run on PayPal's sandbox: the controller planned a ticket, found the right customer by email, the billing worker issued a $49 refund, and the verifier read PayPal back and saw it completed. I also ran a small experiment on the earlier version, training a simple classifier on a model's inner activations to spot silent failures. It scored 1.000 on 400 made-up inputs, against 0.533 for a word-pattern baseline and 0.538 for the agent's own stated confidence. That case was built to be easy, so I read it only as a contrast.

Secondary research. The design follows a few 2026 papers. Advani found that agents claim success falsely in up to 48% of failures on one benchmark, and that AI judges catch it at only 0.65 AUROC (a score where 0.5 is a coin flip). Simple deterministic checks did better. Reddy and colleagues showed that checks before an action runs lift success by 12.4 points. Two more, on calibration and on testing under injected faults, shaped how I measured.

How I tested it

A real payment system doesn't fail on command, so I injected faults. In the sweep, a fault injector made tools misbehave on 30% of calls: accept a refund and then go quiet, say "saved" while writing to the wrong place. Then I scored the verifier against the hidden truth.

Over 40 tickets, 38 tasks could be checked. The agent reported success on 50.0% and PayPal's record confirmed 44.7%. The silent-failure rate was 15.8%, which is 3 of the 19 claimed successes. The verifier caught those 3, and one more slipped past it: a ticket where two customers share a name. The controller can pick the wrong one, and the check still passes, because the action did happen. I count that as a blind spot, not a footnote. The verifier also raised one false alarm.

Be careful with these numbers. It's one sweep of 40 tickets with no error bars. The earlier version, on a different model, measured 4.8% in a similar test, and 25% on an even smaller one, so the rate depends on the model and the workload. And the sweep used injected faults, not live PayPal for every ticket. Only one refund ran live.

There are 59 automated tests that need no network or account. They cover the verifier, the rule check and the metrics arithmetic.

What I learned

  • A confident "done" is a claim. Check it against the record the money lives in.
  • A retry once double-paid a refund in the earlier version, and both attempts looked fine at the end. The fix is a guard that remembers each call and replays the old result instead of running it twice, plus a total limit per order.
  • A bug in my own scoring. I'd treated the agent's confidence as the chance it succeeded, but a worker that says "blocked" at 0.9 is confident about failure. One line changed the calibration numbers a lot.
  • Publish the blind spot. A checker that hides its limits asks for the trust it tells you not to give an agent.

Links

Available · London / UK · senior AI, AI FDE, contract

Building something with AI?

Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.

or leave your email

Replies within ~24 hours · London, UK · on-site, hybrid or remote