README.md main
cifailinglicenseMITsetup2 min

An AI teammate that fixes flaky tests before standup.

Patchwork watches your CI, reruns a failing test in a sandbox until it repeats, finds the root cause and opens a small pull request with the fix. You review it like any other change.

Install on GitHub
  • Free for open source
  • No code used for training
  • 2 min setup

CI · acme/checkout-api

50 runs
test_refund_partialflaky

Failed in 3 of the last 50 runs on main. Passes on re-run.

All workflows · main

  1. fix: currency rounding on partial refundsCI #4182 · test_refund_partial failed02:09
    4m 12s
  2. chore: bump pytest to 8.3CI #4181 · 1,284 passed01:47
    3m 58s
  3. feat: refund reasons in CSV exportCI #4180 · 1,284 passed00:52
    4m 05s
  4. fix: ledger export timezoneCI #4171 · test_refund_partial failedMon
    4m 20s
  5. refactor: split refund queueCI #4153 · test_refund_partial failedFri
    4m 16s

Sandbox

py 3.12
test_refund_partialIsolated copy of CI #4182 · 10 workers
0 / 50 runsNot started
Every hollow cell is a failed run.

Trace · 6 failed runs

pytest
run 07run 13run 22run 29run 37run 44
FAILED tests/test_refund.py::test_refund_partial
tests/test_refund.py:33
assert order.balance == Decimal("40.00")
AssertionError: expected 40.00, got 100.00
Same interleaving in 6 of 6 failures
refunds/queue.py:88 in flush()
refunds/queue.py:61 in _drain()
ledger/write.py:142 in commit()
balance read before lock

Root cause. flush() reads the order balance while a ledger write for the same order is still open, so the refund lands on a stale balance.

Fix ready: 2 files, +14 -6
Pull request #918 +14-6

Fix race in refund flush #918

Openpatchwork wants to merge 1 commit into main from fix/refund-flush-race
patchwork opened this at 02:13

test_refund_partial failed in 3 of the last 50 CI runs. In a sandbox it failed 6 of 50, every time flush() overlapped an open ledger write for the same order.

The balance is now read inside the per-order lock, and _drain() waits for that lock instead of skipping. No change for single refunds. 500 sandbox runs after the change: 0 failures (+0.4s).

ledger/write.py+10 -3
@@ -141,6 +141,13 @@ class Ledger:
141141 def commit(self, order_id, amount):
142- balance = self.read(order_id)
143- self.lock(order_id)
144- self.write(order_id, balance - amount)
142+ with self.lock(order_id):
143+ balance = self.read(order_id)
144+ self.write(order_id, balance - amount)
145+ self._open.discard(order_id)
146+ self._done.set(order_id)
Omar Haddad 02:31

Reading inside the lock fixes it. Does wait_for() need a timeout, or can it just block?

patchwork 02:31

The 2 s timeout stops one stuck write from holding the whole queue. It raises LedgerTimeout, which your retry job already handles.

145147 self.emit("ledger.committed", order_id)
146148
149+ def wait_for(self, order_id, timeout=2.0):
150+ """Block until no write is open for this order."""
151+ if order_id not in self._open:
152+ return True
153+ return self._done.wait(order_id, timeout)
refunds/queue.py+4 -3
@@ -57,6 +57,7 @@ class RefundQueue:
5757 def _drain(self):
5858 for order in list(self.pending):
59- if self.ledger.is_open(order.id):
60- continue
61- self.ledger.commit(order.id, order.amount)
59+ # wait instead of skipping, so no refund reads a balance mid-write
60+ if not self.ledger.wait_for(order.id):
61+ raise LedgerTimeout(order.id)
62+ self.ledger.commit(order.id, order.amount)
6263 self.pending.clear()

Camera · on call

02:13
Live

Photos · last night

1 of 3
Hands typing on a laptop full of green terminal output

01:58 Re-run number four

Notes

4 notes

Flaky tests cost more than they look.

Every re-run is ten minutes of someone's focus, and every ignored failure teaches the team that red builds are normal.

41%of CI failures in a typical monorepo are flaky, not real
6 minmedian time from a failed run to a fix pull request
0merges without a human approving the change

Free for open source.

Install the GitHub app, pick repositories, and Patchwork starts with the next failing run. Setup takes about two minutes.

Open sourcePublic repositories, unlimited fixes$0
TeamPrivate repositories, all CI providers, Slack alerts$19 /dev/mo
EnterpriseSelf-hosted sandboxes, SSO, audit logCustom

Nothing merges without you.

  • No code used for trainingYour code is never used to train models, ours or anyone's.
  • Sandboxes are thrown awayThey run in your region and are destroyed after each job.
  • Pull requests onlyPatchwork never pushes to protected branches. Your review rules apply.

Questions engineers ask.

Does Patchwork see my whole codebase?

It reads the files involved in a failing test and their direct dependencies, inside a sandbox in your region. Nothing is kept after the job ends.

Can it break my build?

No. It only opens pull requests, which run your full checks and need your approval like any other change.

What if the flake is not in my code?

When the cause is shared infrastructure, such as a slow test database, it labels the failure, shows the evidence and tags the owner instead of writing a fix.

Which languages work?

Python, TypeScript, Go, Java and Ruby today, with Rust and C# in beta.

Which CI providers?

GitHub Actions on the free plan. GitLab CI, CircleCI and Buildkite on Team.

02:09. A test went red. Reproduce it from CI.