Without history, an AI agent got 27% right (1,886 lines)
We tested 19 AI models in 136 setups on real bank lines from ten Cypriot businesses. The company’s own history moved the score far more than the model did.
By Christopher DosinFounder / CTO
Updated 7 min read
In this article
- 5 charts
- 1 table
- 3 key figures
Contents
- History took the same agent from 27.0% to 94.8%
- Account choice is convention: 90.0% against 41.7%
- The Cybooks Brain: 72.6%, no significant difference from the strongest agent
- On 1,000 fresh lines, every system landed inside its interval
- A bigger model: nine times the cost for 1.6 points less
- How we measured
- Final thoughts
“It knows accounting.” Most AI models do.
But does it know how your company books its payments?
- The AI agent, without the company’s history
- 27.0%
- The AI agent, without the company’s history
- The same agent, with three months of history
- 94.8%
- The same agent, with three months of history
- The Cybooks Brain’s memory step, no AI call
- 96.9%
- The Cybooks Brain’s memory step, no AI call
We ran 19 models in 136 setups through the Cybooks bank line benchmark. It asks every system one question: given a bank line, which account does it belong to? The lines are real. They come from ten Cypriot businesses, and from one European e-commerce brand that brought its past books.
Key takeaways
- History beats the model. The same AI agent was right on 27.0% of lines without the company’s history and 94.8% with it.
- No significant difference, and cheaper. On 1,500 unseen lines the Cybooks Brain was right on 72.6%, the strongest agent alone on 73.5%. It cost $4.28 per 1,000 lines against $11.19.
- It held on fresh lines. Every system ran once more on 1,000 lines no system had seen, and each landed inside its interval.
- Cost moves by multiples. Claude Opus 5.5 at max cost about nine times GPT-6.1 Terra at high, for 1.6 points less.
- Search adds little. Web search added about one point, within the noise.
- Memory needs no AI call. The Cybooks Brain’s memory step was right on 96.9% of the lines a bank payment can be booked to.
On the e-commerce brand’s lines, an AI agent built on GPT-6.1 Terra was right on 27.0% when it saw none of the company’s earlier bookings. Reading the first three months of bookings, the same agent was right on 94.8%.
- AI agent without historyGPT-6.1 Terra27.0%
- AI single call without historyGPT-6 Luna31.3%
- The same AI agent with historyGPT-6.1 Terra94.8%
- Cybooks Brain with three months of historymemory step, no AI call96.9%
Data: Cybooks bank line benchmark, one company’s past books, November 2026cybooks.com.cy/blog/ai-bank-lines-without-company-history
Both setups without history picked from the company’s own 29 accounts, so a real new company would likely score lower.
The company’s history moved the same agent from 27.0% to 94.8%.
Two accountants can book the same payment to different accounts and both be defensible. What matters to a business is that a payment is booked the way it was booked before. A model that knows accounting but not the business picks the textbook account.
- Booked to that account before989 lines90.0%
- Not booked to that account before511 lines41.7%
Data: Cybooks bank line benchmark, 1,500 held-out lines from ten businesses, November 2026cybooks.com.cy/blog/ai-bank-lines-without-company-history
History decides: 90.0% where the account was known, 41.7% where it was not.
Every system ran once on 1,500 bank lines, 150 from each of ten businesses, and no setting was chosen on these lines. A line with no answer counts as wrong. Costs are US dollars at the model providers’ list prices.
- Cybooks Brain, agent step only$6.28 per 1,000 lines73.5%
- GPT-6.1 Sol agent (xhigh)$11.19 per 1,000 lines73.5%
- Cybooks Brain, people rule off$4.45 per 1,000 lines73.2%
- GPT-6.1 Terra agent (high)$7.98 per 1,000 lines73.2%
- Cybooks Brain, as configured in Cybooks$4.28 per 1,000 lines72.6%
- GPT-6 Luna agent (xhigh)$1.67 per 1,000 lines68.9%
- Claude Opus 5.5 single call (max)$17.50 per 1,000 lines58.1%
- Cybooks Brain, memory step only$0.00, no AI call37.5%
Data: Cybooks bank line benchmark, 1,500 held-out lines from ten businesses, November 2026cybooks.com.cy/blog/ai-bank-lines-without-company-history
The Cybooks Brain did not beat the agents. As configured in Cybooks, lines paid to people are never sent to the AI (the people rule). That version is 0.93 points below the strongest agent, a difference the test cannot tell from noise, and it costs less: its memory step answers 26.8% of lines with no AI call, which saves 268 agent calls in every 1,000 lines.
The rule has a price. It puts the Cybooks Brain 0.93 points below its own agent step alone, and it saves $2.00 per 1,000 lines.
We wrote down a bar before the run: at most 1 point below the strongest agent. It was written for the version without the people rule. The interval around that difference reaches 1.60 points, and 2.20 points as configured, so the Cybooks Brain did not meet that bar.
No significant difference, at $4.28 per 1,000 lines against $11.19.
The 1,500 lines shaped the paper. So every system ran once more on 1,000 new lines from the same ten businesses, lines no system had seen, picked by a rule fixed before any run.
- GPT-6.1 Sol agent (xhigh)73.5% on the 1,50071.9%
- Cybooks Brain, agent step only73.5% on the 1,50071.8%
- Cybooks Brain, as configured in Cybooks72.6% on the 1,50071.1%
- GPT-6.1 Terra agent (high)73.2% on the 1,50071.1%
- GPT-6 Luna agent (xhigh)68.9% on the 1,50067.5%
- Claude Opus 5.5 single call (max)58.1% on the 1,50059.2%
Data: Cybooks bank line benchmark, 1,000 fresh lines from ten businesses, November 2026cybooks.com.cy/blog/ai-bank-lines-without-company-history
We also wrote down 15 claims before that run. Nine held on both readings, four on one reading, and two did not hold.
The order of the systems is the same on fresh lines.
Among the agent setups we screened, accuracy differs by points and cost by multiples.
- Frontier
- Other models
- Open-weight
| Model | Provider | Licence | Lines right | Cost per 1,000 lines | On the frontier |
|---|---|---|---|---|---|
| gpt-oss-120b | OpenAI | Open-weight | 54.4% | $0.58 | Yes |
| GPT-6 Luna | OpenAI | Closed | 64.4% | $0.59 | Yes |
| DeepSeek V4 Flash 0731 | DeepSeek | Open-weight | 64.6% | $0.94 | Yes |
| GLM-5.3 Flash | Z.ai | Open-weight | 64.2% | $1.52 | No |
| Nemotron 3.5 Super VL 120B | NVIDIA | Open-weight | 62.6% | $1.95 | No |
| Claude Haiku 5.5 | Anthropic | Closed | 66.8% | $2.15 | Yes |
| DeepSeek V4.1 Flash | DeepSeek | Open-weight | 63.8% | $3.18 | No |
| Beam 1.0 | Reflection | Open-weight | 53.8% | $5.02 | No |
| Nemotron 3 Ultra 550B | NVIDIA | Open-weight | 44.2% | $5.50 | No |
| GPT-5.4 mini | OpenAI | Closed | 62.6% | $5.87 | No |
| GPT-6.1 Terra | OpenAI | Closed | 70.2% | $5.99 | Yes |
| GLM-5.2 | Z.ai | Open-weight | 64.0% | $8.82 | No |
| GPT-6.1 Sol | OpenAI | Closed | 70.8% | $10.33 | Yes |
| DeepSeek V4 Pro 0813 | DeepSeek | Open-weight | 64.2% | $12.09 | No |
| Claude Sonnet 5.5 | Anthropic | Closed | 66.4% | $13.62 | No |
| GLM-5.3 | Z.ai | Open-weight | 62.8% | $14.17 | No |
| Muse Spark (open) | Meta | Open-weight | 53.4% | $17.95 | No |
| Kimi K3 | Moonshot AI | Open-weight | 62.2% | $24.27 | No |
| Claude Opus 5.5 | Anthropic | Closed | 68.6% | $54.31 | No |
The shaded zones split the 19 models at their median lines right and median cost: upper left is more accurate and cheaper than the median, lower right is less accurate and dearer. The violet line is the frontier: the models nothing else beats on both. Lines right starts at 40%, just below the lowest score. Anthropic at Batch API prices, 50% of list; the others at synchronous list prices.
Data: Cybooks bank line benchmark, 500 screening lines, November 2026
GPT-6.1 Terra as an agent at high cost $5.99 per 1,000 lines for 70.2% right; Claude Opus 5.5 as an agent at max cost $54.31, at the Batch API price, for 68.6%.
Web search is the other thing money buys. Averaged over two runs each, the agent was right on 70.4% of lines with web search and on 69.2% without it.
Pay for history before you pay for a bigger model.
We drew 2,000 bank lines from ten Cypriot businesses, 200 from each, and labelled each with the account a signed-in person booked on it.
| Line set | Lines | What it is for |
|---|---|---|
| Screening | 500 | Choosing the setups to test, 50 from each business |
| Held out | 1,500 | The final comparison, 150 from each business, unseen before the run |
| Fresh | 1,000 | The re-check: every system once more, on lines no system had seen |
| Case study | 6,487 | One European e-commerce brand’s April to June lines |
A system sees only what the person could have seen: bookings of lines dated at least a day earlier and made at least 60 seconds before the person booked the line. The labels were not reviewed by an accountant, so some errors are label noise.
A line is right when the answer exactly equals the account the business booked. No answer counts as wrong, and no AI grades the answers. Intervals are Wilson 95% intervals, and two systems are compared with an exact McNemar test on the lines where they differ.
- The case study’s main figures use the line texts in the company’s books, and cover the account only, not the contact or the VAT.
- On the 2,114 lines we could check on real bank-feed text, 83.4% were right just after the import against 94.7% with the books’ text, and 93.0% once the feed’s own history built up.
- The AI setups without history picked from the company’s own 29 accounts. A real new company would likely score lower.
- Every system ran once on each line set.
- Cybooks does not book lines on its own. It fills in the account and a person confirms.
A strong model knows accounting. It does not know your company.
On these lines, a company’s own history moved the same agent from 27.0% to 94.8%, and a memory of its past bookings reached 96.9% without an AI call.
If you compare tools that book your bank lines, ask what they learn from your own books first.
Model names are trade marks of their owners. Cybooks is not affiliated with any model provider.