Superforecasting: The Art and Science of Prediction
by Philip E. Tetlock and Dan Gardner (2015)
What is the likelihood that Donald Trump will be elected president next November? This is the test question I contrived in order to
examine my understanding of this book, which demonstrates how the world's best "superforecasters" use numeric and non-numeric techniques
to produce results that are often strikingly accurate.
A good deal of the book is based on Tetlock's multi-year Good Judgment Project, which seeks to discover who makes the best
forecasts and under what circumstances. One interesting result, which might well be applied to a judgment of Ben Carson's bid for president,
is that experts forecast quite poorly when venturing outside their areas of expertise. Non-experts seem to fare far better because, out
of awareness for their lack of expertise, they dig deeper and more thoroughly to challenge pre-conceived notions and conventional wisdom.
Teams do even better when the members are diverse enough to provide complementary skill sets.
Much of the numeracy is based on a sound understanding of probability, in particular Bayes' Theorem, which allows us to parcel
a probability computation into manageable pieces. For instance, take my test question. The first thing we know is that
in order to be elected president, Trump must receive the Republican nomination. Thus his chance of being elected president is equal to
his chance of receiving the nomination, multiplied by his chance of winning the fall election.
But the chance of his winning the fall election depends on his opponent; for simplicity we will treat it as a purely Democrat-Republican contest.
So Trump's chance of winning in the fall is equal to the sum of these three numbers: the probability of Clinton receiving the nomination
multiplied by the probability of beating Clinton; the probability of Sanders receiving the nomination multiplied by the probability of
beating Sanders; and the probability of O'Malley receiving the nomination multiplied by the probability of beating O'Malley.
If you compute the same probability for a Democrat it is messier because there are so many more Republican candidates to account for,
but the principle remains the same. Tetlock maintains that many of the best forecasters do not necessarily make explicit computations,
as I have for the leading presidential candidates, but they are familiar with the principles and take them into account.
The book has powerful implications for those who wish to learn to tighten their predictions, to question their assumptions and to measure results.
How do you test predictions? How do you measure them? Although the book is all about numbers and how they are used, it is light on number
crunching; most data is represented graphically, and I recall finding only one equation, which demonstrates Bayes' Theorem, a perfect lead-in
for my test question.
You can measure my progress on the question at
http://www.dalesdemocracy.com, which I expanded to include the top six presidential
candidates; there is insufficient data to cover more. And true to the superforecasters' best practices, I am adjusting the data and
the methodology as the campaign progresses. But don't wait to see how close my predictions get to the final results--read the book
and make up your own test question. Three and a half stars.