“Prediction is very difficult, especially about the future.” This quote, attributed to several personalities including Niels Bohr and Yogi Berra, seems to sum up what we all would attest to know. However, just as the cost of living has not affected its popularity, the necessity to conjecture is unavoidable. Humans are constantly processing information and developing correlations and hypotheses to explain the world around them, yet we are surprisingly bad at it.
In Nassim Nicholas Taleb’s book The Black Swan, he preaches the foibles of speculation and cautions that we need to hedge our interests against the unexpected. He uses black swans as an example since most human beings historically would be quick to postulate that all swans are white, and they might even be willing to accept financial risk to back such a claim given that their extensive experience seems to prove their theory. Central to this weakness in human nature is what Taleb calls the confirmation bias. This is the tendency we have to formulate a hypothesis, and then look for evidence that proves our theory, rather than look for examples that disprove it. Several bloggers have reviewed this book (see for example Think Inside the Cube, Panarama of the Mountains, and Quasi-Coherent), so be sure to read what they have to say. What made this book so important to me, though, are the extensive examples of real world situations and vast bibliography of institutional experiments that illustrate how wrong our instincts can be.
The need to conjecture seems rooted in the fact that we have limited resources available to gather information, store it, and retrieve it. To facilitate this process, humans attempt to compress the data by recognizing patterns, rather than trying to absorb every digit of detail. This leads to the narrative fallacy, in Taleb’s words, where we try to create a coherent story to summarize our observations. For example, if a child is shown a photograph of a person and told that he is a member of a larger group, and then asked to describe the other members of the group, he is likely to get some things right, but fail in other areas. Typically, children would assume that the other members of the group would share the same language, which is reasonable, and somehow the children would avoid the assumption that all members were the same height and weight. However, the children would often predict that all members shared the same skin color, which was a brash extrapolation. Studies of split-brain patients, individuals who have had their corpus callosum severed to disconnect the two sides of the brain, indicate that this causality processing resides in the left hemisphere of a right handed person, emphasizing that there are significant biological factors that influence our reasoning. Additionally, patients having higher concentrations of dopamine, which regulates moods and lowers skepticism, are found to be more prone to developing wild extrapolations of patterns.
More troubling than our tendency to formulate and speculate, is our inability to accurately gauge our level of accuracy. For example, a group may be asked to estimate the number of books at their local library. Instead of an exact number, they are asked to provide a range in which the true value resides, within a 98% level of confidence. The estimates, it turns out, are wrong by at least an order of magnitude larger than ideal 2%, indicating that we generally are far too confident in our estimates. Surprisingly, additional experiments showed that the more information we have, the more likely we are to overestimate the accuracy of our predictions. Therefore, more data and more knowledge actually increase the likelihood that you would be wrong since overconfidence would lead you to estimate ranges that were too narrow.
“The future is not what it used to be,” is another quote that applies here, also often attributed to Yogi Berra, but is of dubious origin. So, please, be mindful of your assumptions, particularly when making important life decisions. And, by the way, black swans are native to Australia, where I had the good fortune to see them in the wild when I lived near Melbourne in the mid-90’s.
Sophronismos 