How often our import reads a bank message wrong

Every finance app says its import is accurate. None of them say what they counted, so the claim cannot be checked and does not mean anything. Here is ours, with the method attached.

Two different questions

PocketVault Finance can read the text messages your bank sends and turn them into transactions. When your spending looks wrong, it is because of one of two failures, and they need separate numbers:

1. A message arrived and we read it wrong. Wrong amount, wrong direction, or a transaction recorded from a message that never described one.
2. No message arrived at all. Nothing to read, so nothing appears, and the app has no way of knowing something is missing.

The first is our fault. The second is not, but it is still a hole in your ledger, and you deserve to know how big it is.

What we did

For the first number, we took a real Android inbox — 9,109 bank messages spanning March 2022 to August 2026, across an Indian and a US bank — and drew 500 at random with a fixed seed. Those 500 were read and labelled by hand before the importer was run on them, which matters: if you label after seeing the output, you unconsciously grade on a curve. Of the 500, 116 described a payment.

Five of the 116 came out wrong. That is 4.3%, with a 95% confidence interval of 1.9% to 9.7% — a range that wide is what a sample of 116 buys you, and quoting the 4.3% without it would overstate what we know.

For the second number, there is no sample to draw; you need a source of truth about what actually happened. We used three of our own bank statements and checked them line by line against the same inbox. A statement line counts as covered if a message from that bank, dated within three days, states the same amount.

Eight of 37 statement lines had no message at all. That is 21.6%.

What the failures were

Three of the five were the same shape: a GOOGLE TEMPORARY HOLD for $1.00, recorded as a dollar of spending. A hold of that kind is a card check that gets released a few days later — no money moves. The same shape appears 23 more times in that inbox, so it was not an edge case; it was a recurring dollar of fiction. We wrote up that one separately.

The other two were Axis fee notices that said ₹25 when the statement said ₹29.50, because the message quoted the charge before tax. That one has its own post too.

The eight missing messages were more interesting than we expected. All eight were bank-initiated postings under ₹500 — service charges and small interest credits — while the payments the account holder made themselves did come through. That is 37 lines across three accounts, so treat it as a direction rather than a rule.

It is a direction worth knowing about, though, because it is about to get steeper. From 1 January 2027 the RBI’s alert mandate applies to transactions above ₹500, and some banks already skip small UPI debits today. A ledger built only from text messages will be missing its small entries by design, not by accident — which is why the app lets you bring in a statement as well, and why we reconcile against one.

The number we are not publishing

Both defects above are now fixed, and the same sample scores perfectly afterwards. That figure is worthless and we will not quote it. The sample is where the bugs were found, so it cannot also be the test that they are gone — you would be marking your own homework with the answers in front of you. The next figure has to come from a fresh random draw.

Why any of this is on our website

Because a number that only ever improves is marketing, not measurement. If you check our accuracy page in six months and it has gone up, that is the system working: it means we measured a worse week and published it anyway.

There is a second reason. The app already shows you every message it could not read, in a review queue, instead of quietly dropping it. Publishing the rate is the same commitment at a different scale — you should be able to see what we get wrong without having to catch us at it.

You can check the privacy side of this the same way, and we would rather you did: the ways to verify it yourself are here.

What we will not do

We will not publish the inbox. It is a real person’s bank messages, and no amount of anonymisation makes that safe to hand out. What is published is the method, the counts, and what broke — which is enough for you to run the same measurement on your own inbox and get your own number.

And it will be your own number. Your banks, your alert settings and your phone will not match ours.