The bank line benchmark for real company books

We ran 19 models in 136 setups, each as a single call and as an agent that can read the business’s earlier bookings and search the web, and asked each one question: which account does this bank line belong to?

These are the findings, on the 500 screening lines unless they say otherwise.

Key takeaways

Without imported past books, the Cybooks Brain shows no significant difference from the strongest agent, and costs less. It did not meet the bar we set before the run.

The 1,500 held-out lines shaped the paper, so every system ran once more on 1,000 new lines from the same ten businesses: lines no system had seen, picked by a rule fixed before any run.

On one company’s past books, the same AI agent was right on 27.0% without the company’s history and 94.8% with it. The Cybooks Brain’s memory step would have been right on 96.9%, with no AI call.

A test on the company’s past books: one company, line texts from its books, account only. The memory step is not yet switched on for customers, so nothing was filled in for the company. A sample of 3,004 of its 6,487 test lines, fixed before the run; the AI setups without history picked from the company’s own 29 accounts. The agent with history is the benchmark’s agent with web search, reading every January to March line of the books; the agent configured in Cybooks does not read imported rows, so its figure here would likely be lower.

Across ten young businesses the Cybooks Brain, as configured in Cybooks, is right on 72.6% of the held-out lines, and on 82.0% of the lines of mature ones (businesses with at least three months of bookings by a person and at least 200 earlier bookings). On one company with three months of history, its memory step alone would have been right on 96.9% of the lines a bank payment can be booked to.

The steps are not one controlled experiment. The e-commerce brand is a different dataset: another company, labels from its previous system’s ledger and its own mix of accounts. Steps 1 and 2 include the agent and step 3 does not, and step 3 is a test on past books. They point the same way, but cannot say how much of the rise comes from knowing the business.

On the same company’s April to June 2026 lines, scored on every ledger line: right on 92.2% after one month of history, 94.5% after two and 95.1% after three.

The figures above use the line texts in the company’s books. A bank feed delivers other text, so the same test ran again on the 2,114 lines that matched a row in the company’s own bank exports.

Each business books its own way. A model that cannot see the history cannot learn it.

Businesses differ far more than systems do. On the held-out lines the Terra agent is right on 45% of Business I and 94% of Business J. Business E shows convention: the Opus single call gets 5% there, the agents 87% to 92%. A single call sees no history, so it cannot learn a business’s own convention. Money in stays harder than money out for every system.

More effort mostly means more tool use. The agents gain from it, but the steps are small and the top levels cost far more.

GPT-6.1 Sol as an agent rises from 66.0% at low to 70.8% at xhigh, Claude Opus 5.5 from 65.6% to 68.6% at max, Claude Haiku 5.5 from 57.8% to 66.8%. The top levels cost far more: Opus 5.5 at max uses 2.5 times its tokens at low.

The six setups largely miss the same lines. For every one of them, the largest kind of error is the right account type in another account group.

Real lines from real books, labelled with the account a person booked. Every answer is one account name, compared with the account the business booked.

The task

Given a bank line and the bank account it came through, choose the account for the other side from the business's chart of accounts.

The businesses

Ten businesses in Cyprus, shown only as letters (A, B, E, F, G, H, I, J, M, N). Of the businesses with at least 200 labelled lines, twelve qualified; two were left out because almost all their lines go to one or two accounts.

The lines

A seeded draw of 2,000 bank lines, 200 per business: 500 for screening and tuning, 1,500 held out. The screening set has 447 money-out and 53 money-in lines.

The accounts

Each business’s chart holds about 213 postable accounts. The screening run offered a list of 120 per line, and the booked account was on it for 481 of 500 lines, so 96.2% is the ceiling for every setup.

The leak rule

A system sees only what the person could have seen: bookings of lines dated at least a day earlier and made at least 60 seconds before the person booked the line.

The labels

The account a signed-in person booked on the line in Cybooks. Bookings by rules, bill matching or the AI are not labels. The labels were not reviewed by an accountant, so they carry each business’s own habits.

The case study

A different dataset: one European e-commerce brand that brought its past books from its previous accounting system. The memory step learned from January to March 2026 (8,463 lines) and was tested on April to June 2026 (6,487 lines), in a separate copy of Cybooks. Line texts come from its books; 2,114 lines were checked again on real bank text.

The re-check

Every system ran once more on 1,000 fresh lines from the same ten businesses, picked by a rule fixed before any run: the same label rule, lines no system had ever answered, and a seeded draw spread as evenly over the businesses as the lines allowed.

Every answer takes the same path, from the model’s reply to a line that counts as right or wrong.

Accuracy

Right lines over all lines. A run with no parseable answer counts as wrong.

Intervals

Wilson score intervals at 95%.

Two systems

An exact two-sided McNemar test on the lines where they differ.

Cybooks Brain against an agent

A paired bootstrap 95% interval of the difference: 10,000 resamples of the 1,500 lines, with a seed fixed before scoring.

Cost

Costs are in US$ per 1,000 lines at provider list prices, from recorded tokens, not invoices: Anthropic at Batch API prices, 50% of list, the others at synchronous list prices. The Cybooks Brain is not on the screening chart: its cost was not measured on those lines.

Time

Mean wall-clock seconds per line, provider queueing included. Batch runs record none.

Before the held-out run we wrote down what each system had to reach, and judged each criterion once, on the 1,500 held-out lines. Five passed and three failed.

1. CriterionRule, set before the runResultVerdict

2. CriterionAccuracy of the Cybooks Brain’s agent step

   Rule, set before the runPaired difference to the research build (the Terra agent with web search): bootstrap lower bound at least -3 points

   Result73.5% against 73.2%: +0.33 points (-0.73 to +1.47)

   Pass

3. CriterionAgent alone: the account it picks was booked before on the same side

   Rule, set before the runAt least 78% right, Wilson lower bound at least 75%, on at least 60% of lines

   Result82.0% (79.6 to 84.2) on 72.3% of linesResearch build: 82.4% (80.0 to 84.6) on 71.7% of lines, pass

   Pass

4. CriterionClean answers

   Rule, set before the runNo account outside the list; at most 2% no answer

   Result8 outside the list, refused by Cybooks’ code and counted as no answer; 1.3% no answerResearch build: 0 outside the list, 0.8% no answer, pass

   Pass

5. CriterionSpeed and cost of the agent

   Rule, set before the run90% of lines within 20 s; at most $8 per 1,000 lines with search

   Result10.61 s; $6.28 per 1,000 lines, no web searchOn recorded tokens; 5 time-outs unmetered. Research build: 10.8 s; $7.98, $8.36 with every search uncached, fail

   Pass

6. CriterionCybooks Brain, memory step: confident answers

   Rule, set before the runAt least 95% right, Wilson lower bound at least 92%, on at least 25% of lines (40% for mature businesses)

   Result94.5% (91.9 to 96.4) on 26.8% of lines; 40.5% for mature businessesFresh run: 96.1% (93.3 to 97.8) on 30.9% of lines; 46.7% for mature businesses, pass

   Fail

7. CriterionCybooks Brain against the best agent alone

   Rule, set before the runPaired difference: bootstrap lower bound at least -1 point

   Result-0.33 points (-1.60 to +0.93)Same design with the research build of the agent: -0.27 points (-1.40 to +0.87), fail

   Fail

8. CriterionBooking without a person

   Rule, set before the runAt least 363 lines; account, contact and VAT all right at least 97% of the time, Wilson lower bound at least 95%

   Result136 lines; 87.5% (80.9 to 92.0)Same design with the research build of the agent: 135 lines; 87.4% (80.8 to 92.0), fail

   Fail

9. CriterionCost of Cybooks Brain

   Rule, set before the runAt least 250 agent calls saved per 1,000 lines; cost at most 75% of the agent alone

   Result268 saved (246 to 291); cost 70.8% on model tokens, no web searchSame design with the research build of the agent: 69.6% on model tokens, 75.03% with every search uncached

   Pass

10. CriterionAgent alone: the right account among three shown

    Rule, set before the runSet after a top-3 experiment

    ResultNot measurable: the research build gives one answer and the agent step one answer and one alternative

    Not measured

So Cybooks does not book lines on its own yet. As designed, the system fills lines in and a person confirms them. Cybooks Brain in this table is the version without the rule that lines paid to people are never sent to the AI, for which the criteria were written. Agent step: the Cybooks Brain’s agent, the agent configured in Cybooks. Research build: the GPT-6.1 Terra agent with web search, judged on the same design.

• Ten businesses. The held-out set has 1,500 lines, 150 per business, so per-business numbers carry wide intervals.
• Labels as booked. Each label is a single booking by a person, not reviewed by an accountant. Some errors are label noise.
• One company, a test on past books. The case study is one European e-commerce brand, run through a separate copy of Cybooks while the memory step is not yet switched on for customers. It covers the account only, with labels and line texts from the company’s books. On raw bank-feed text right after the import fewer lines are right: 83.4% against 94.7% on the lines that could be checked. It is a different dataset from the held-out lines.
• Books that repeat. A plain lookup on the opening two words of the line text is right on 97.2% of this company’s lines, and on 33.8% of the screening lines of the ten benchmark businesses. History helps this much here because this company’s books repeat a lot.
• Run once, not in production. Every system ran once on each line set. The Cybooks Brain’s agent step read each line’s history from an export of the same data.
• One bar, borderline. The memory step’s confident answers were right on 94.5% of the lines it filled in on the held-out set, just under the bar set before the run, and on 96.1% on the fresh lines, which meets it. Read together: about 95%.
• Time and money, roughly. Time per line mixes model speed with provider queueing, and is missing for Anthropic. Costs come from list prices, not invoices.
• Not in this benchmark. Gemini, GPT-6 Astra, Claude Fable 5.1, Muse Glimmer 30B and a human baseline.

Model names are trade marks of their owners. Cybooks is not affiliated with any model provider.

Version 1.0, 11 Nov 2026

Models

19 models and 136 setups: each model at every reasoning level it supports, as a single call and as an agent. Every setup ran once.

Line sets

500 screening lines, 1,500 held-out lines and 1,000 fresh lines from ten businesses, and one company’s 6,487 test lines for the case study.

Scoring

Exact match with the booked account, Wilson intervals, exact McNemar tests and a paired bootstrap. The method was frozen on 2 November 2026, 12:00 EET, before the screening run.

Re-check

Every system ran once more on 1,000 fresh lines no system had seen. Of 15 claims written down before that run, 9 held on both readings, 4 on one reading and 2 did not hold.