A few days ago I posted a problem cited in Jason Zweig’s Your Money and Your Brain. The problem made no sense to me, and to various commenters on the blog. I can now reveal where we (and the writers of the problem) went wrong.

I’ve had a very helpful email conversation, first with Jason, then with Robin Hogarth, the academic who first used the problem on hapless stats experts at the University of London, who was kind enough to answer my query about the paper, which he wrote 30 years ago. Here is the problem as phrased in the original paper:

‘It is claimed that when a particular consultant says the market will rise (i.e., a favorable report), it always does rise.’

Subjects were then asked the following question: What is the minimum evidence you would need to check the consultant’s claim,

The paper states that the (obviously?) correct answer is 1 and 4, but many statisticians came up with the ‘wrong’ answer of 1 only, which I also thought was correct.

It turns out that the original problem is not stated fully in the paper (it does say that there was considerable verbal explanation accompanying it, which may have covered the missing information). As it reads, there is no suggestion that you can’t monitor the results of every prediction made by the consultant. But that isn’t what was intended to be offered.

The choice was supposed to be that you could check one instance of any or all of the checks 1 to 4. These would be sampled from the consultant’s work. What the question really should have asked was not if you could check that the consultant’s claim was true – clearly you could never prove it to be true from this sampling – but rather which of the four data would you need to check to make sure you had no evidence that countered his claim.

Then, finally, the 1 and 4 answer is correct. But as stated in the original paper, the problem isn’t effective – and that means poor old Jason, in using this example, was bound to cause confusion. He has tried to correct his book by slightly changing the wording in a later edition, but in fact the conditions to make the problem work are so messy that it just doesn’t provide an example of what he was trying to show. That’ll teach him to depend on an academic paper!

Seriously, it would be highly unfair to single out Robin Hogarth’s thirty-year-old paper, but this does illustrate once again a topic often discussed on Nature Network – there is so much that could be done to improve the quality of writing in papers. As demonstrated here, this doesn’t just impact on whether or not the paper will send you to sleep, it also can influence whether the information from the paper can be properly used.