Dart-throwing experts
These are ultimately unpredictable times, yet we still need to make decisions, which means we still need forecasts. How can we make them better? For one thing, practice disagreement.
“Laws are like sausages, it is better not to see them being made,” said Otto von Bismarck a long time ago. The current pandemic may as well add “science” or “evidence” to the list. All the science that has been going on behind the scenes for decades has suddenly been exposed to the public. Long chains of premises and conclusions that throw an elaborate theory out of the window when one of the assumptions turns out to be wrong, new pieces of research turning previous findings on their head, models that are now spectacularly imprecise because they didn’t take into account one little factor that was unknown at the time — all combine to look like complete chaos. In the course of a few months, policy recommendations have gone from “no need to panic, just wash your hands” to unprecedented lockdowns all across the world — all of which were claimed to be based on scientific evidence.
The dirtiest part of academic work is predictions. Academics love making them, the media love publishing them, yet, in fact, on average they are shockingly imprecise. So much so that we might as well let a dart-throwing chimpanzee decide.
That vivid comparison belongs to the political scientist Philip Tetlock, the first researcher to study the precision of forecasts made by academic experts, on real-life events. His research is truly stunning in scope — in 1984 he gathered predictions about the economy, election results, military conflicts and other geopolitical events from a pool of distinguished American scholars. As the events they were asked to predict were real, not invented, only time would tell if they were wrong or right. And time did tell when, in 2004, Tetlock analysed the results.
It turned out that experts were fairly good at predicting in the short term (about a year on from the moment of asking), but with more distant events (three years or more) the precision of their forecasts amounted to no more than that of a random guess. The only thing that generally saves the experts from professional humiliation is that the media do not seem to care and hardly ever expose the falsity of previously published predictions after the fact.
But what about those of us who do care about the quality of predictions? After all, we still need them even in the most predictable of times, as they underpin every decision we make, from visiting a grocery store to bringing a child into this world. If you happen to be leading a business, your job may require decision making that ranges from the mundane to the far-reaching. Future forecasts that are more or less reliable are absolutely essential to making a good judgement.
If experts in the media are this bad, where does one find better predictions? Look for superforecasters. There are some people, as Tetlock found out, who are far better than others at forecasting the future, even a quite distant one. These people became his primary research subject. In 2011, Tetlock, together with his colleagues, started a forecasting contest that allowed anyone to compete in the precision of their forecasts. Over the years he picked out people who were astonishingly and consistently good — to the point that it looks like magic, yet there is nothing mystical about them.
In fact, they are what we usually call “ordinary people”: retired engineers, teachers, artists etc. They have no access to classified information or sophisticated prediction models. But they do have a certain set of thinking habits, skills and cognitive traits, described in detail in the 2016 book.
It takes a whole book to go through all of the things that make superforecasters superior, so for this post, let’s pick one thing that can be most easily integrated into every person’s life: celebrating contradictions and disagreements.
Superforecasters arrive at their precise predictions by evaluating contradictory evidence, doubting themselves and changing their minds where necessary. Dealing with contradictions is harder than it sounds — who wants to live in conflict?
Who wants an argument at the dinner table? In normal life we tend to do everything we can to avoid clashes of opinions. We tend to agree easily. This is why we also tend to make bad judgements.
However natural this conflict-averse behaviour may be, expressing and evaluating contradictory arguments is what drives good decision making. Superforecasters have learned to work through these evidence battles solo, while for the rest of us the easiest way forward is to collectivise this thinking process by encouraging the expression of competing arguments.
It’s not only families who fall into the trap of agreement. Business leaders are the worst offenders. The bigger the ego of the boss, the more explicit their tendency to gather only “blue teams” that consist of loyalists and yes-men, ready to provide all kinds of “proof” for the idea the boss is already leaning towards. Entrepreneurs are even ready to spend money on many hours’ worth of expensive outside consultants to doubt every idea that might contradict their previous beliefs.
Want fewer mistakes? Create a “red team” as well (even one “red adviser” would do) and give them the job (and the right!) to doubt your assumptions and arguments. The road to a decision will become much longer and bumpier, yet it will lead you in the end to a better place.
