# cfo-bench breakdown and approval gate (Agents Finance, 3 November 2026)

Files that go with the Agents Finance entry on cfo-bench:
https://agentsfinance.io/entries/resource-cfo-bench-ai-bookkeeping-benchmark-claude-code-codex-2026/
(Spanish: https://agentsfinance.io/es/entradas/recurso-cfo-bench-benchmark-contabilidad-agentes-ia-claude-code-codex-2026/)

cfo-bench is Tom Eberle's open benchmark of AI agents doing the books of a fictional French
company: https://github.com/tom-eberle/cfo-bench (MIT). These files are **Agents Finance
templates based on it**; they are not published by its author. Our own code is MIT-0.

| File | What it is |
|---|---|
| `cfobench_breakdown.py` | Reads cfo-bench's public `results/results.jsonl` and writes the three CSVs below. Python 3.9+, standard library only. |
| `by_task.csv` | Our output: tries done right per agent (out of 3) for each of the 36 requests. |
| `by_family.csv` | Our output: success, safe tries and API-equivalent cost per request done right, per agent and request type. |
| `unsafe.csv` | Our output: the 37 unsafe tries, with the harm kinds the graders recorded. |
| `approval_gate.py` | Checks an agent's plan and your draft reply before you send it: every open question needs its own `Q<n>:` answer, a filed period needs `FILED PERIOD OK: <period>`. Exit 0 = PASS, 2 = BLOCK. |
| `approval-gate-kit.zip` | The gate, its test (`test_approval_gate.py`) and 12 example plans and replies. |
| `agents-approval-rules.en.md` / `agents-approval-rules.es.md` | A block of approval rules to paste into AGENTS.md or CLAUDE.md. |

## How we produced and tested them (3 November 2026, Atlantic/Canary)

- cfo-bench at commit `a791fd0d3dc98e71c1fab4327b32ea259ed8dbf6` (release 0.3.1 of 30 October 2026 plus a README chart).
- `cfobench_breakdown.py results/results.jsonl --out out` with Python 3.13: 626 rows read, 540 scored. Its headline
  numbers match `results/summary.md` (success 87/82/68/60/60 %, safe tries 90/95/95/94/92 %).
  `cfo-bench report` run on the same file gave a `summary.csv` byte-identical to the published one.
- `test_approval_gate.py`: 19 of 19 cases as expected (generic OK in English, French and Spanish blocked on
  four kinds of plan, explicit answers passed, a plan with no question passed).
- The CSVs derive from cfo-bench's MIT data (Copyright (c) 2026 Tom Eberle); the licence travels in the ZIP
  as `examples/LICENSE-cfo-bench.txt`. `examples/cfobench-transfer-proposal.txt` is copied unchanged from
  cfo-bench's `tests/fixtures/codex-trap-ambiguous-r1-proposal.txt`. Every other example is fictitious and
  written by us.

## Limits

- The gate uses text heuristics in English, French and Spanish. It errs on the side of blocking and does not
  understand the plan: read the plan anyway.
- The benchmark results are from cfo-bench's pre-release version (runs of 20 and 21 October 2026), with French
  VAT rules and one fictional company. Costs are API-equivalent, as reported by cfo-bench.
- We did not run any agent: we re-computed tables from the published results.