A friend of mine was expecting a result from a medical test, and wanting if necessary to cushion the blow, I prepared the statistical defence to bad test results. As it happens it wasn’t needed, but it’s worth repeating for those who haven’t heard it.

Say the test is 98% accurate for false positives. So only 2% of patients are told there’s a problem when there isn’t. What’s the chance, if you get a positive test result, that you have the medical problem being tested for? The vast majority of people would answer 98%. It’s practically certain. (I know most of the people reading this wouldn’t answer this way, but bear with me.) In fact it’s quite possible that you are more likely not to have the problem than to have it.

Say the test is administered to 1 million people a year, of whom 1 in 1,000 have the medical problem that is being tested for. We’ll also assume a 98% accuracy for false negatives, to keep things simple.

So 980 people who are told they have the problem really have it. But 2% of the million people tested – 20,000 people – will be told they have a problem when they don’t. So with these particular numbers, with a 98% accurate test, the chances are 20:1 against you actually having the medical problem if you test positive.

Now the point of the post is this. I understand statistics fairly well, even though my Masters in Operational Research is now decidely rusty. But I still find the result above surprising and counter-intuitive. What chance have those without training in crunching the numbers?

We are, as human beings, programmed to be bad at understanding probability and statistics. (If you don’t believe this, take a step into any casino.) It’s not that the maths is particularly difficult, but rather that we are pattern seekers and story tellers. We understand the world through patterns, so we see patterns where they don’t exist. This distorts our interpretation of what should be straight-forward statistical results.

Equally, we gain much of our understanding of the world through stories. But stories are specific cases, and often run counter to statistical reality. (How often have you heard an old person on the radio or on TV say ‘I smoked 40 cigarettes a day and it never hurt me’ or ‘the secret of my long life is covering my meals with lots of salt’ or whatever.)

It’s not enough, then, to improve the way we educate people in the maths behind probabilty and statistics, we ought to be aware that there will always be a problem with understanding, and make sure we include a clear explanation accompanying any statistically-based information we present to the public. Do results of medical tests include the simple insights into their reliability mentioned above? I doubt it. But maybe they should.