B.I – Chasing significance and selective reporting

The New Yorker published a lengthy editorial on how natural human behavior affects the scientific community when it comes to studies, in that selective reporting and significance chasing leads to ‘publishing bias’.

“the act of measurement is going to be vulnerable to all sorts of perception biases. That’s just the way human beings work.”

“researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance – the ninety-five-per-cent boundary invented by Ronald Fisher. The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,”

“…the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for.”

“The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe…”

Although the context is regarding the scientific community, I was thinking how there are direct parallels to the business and B.I community as well. perhaps we’re too focused on the pursuit of truth, when the ‘reality’ is that there no truth, only perception & interpretation.

So instead of using data to prove things, we need to look at data to be more of a guide. For example monitoring trends, inflection points, velocity of change over specific numbers. Not that specific numbers are unimportant (e.g. shareholders will continued to demand exact profitability statements), but in terms of managing a business not to exhaust enormous efforts on proving perceptions/suspicions/hypotheses but to define data driven decision supporting guides.