openpolitics.com

If you were running a forecasting tournament over an extended period of time and you had, say, 500-plus questions and thousands of forecasters, and you have estimates of diversity and accuracy over long periods of time, you can work out algorithms that do a better job of distilling the wisdom of the crowd than, say, simple averaging. It sounds risky, but it’s an algorithm known as extremizing, and it works out pretty well.

.. imagine another situation: President Obama is sitting down with friends and they’re relaxing and watching a March Madness basketball game. He’s a fan of March Madness. There is going to be a game between Duke University and Ohio State, and the people around him make estimates on the probability of Duke winning. The estimates are exactly like the estimates we got on Osama; they start around 35 percent, they go up to 95 percent with the center of gravity around 75 percent.

Do you think when his buddies offered these odds estimates, President Obama would have said, “Sounds like 50-50 to me”? Or would he have said something like, “Sounds like three-to-one favoring Duke”?

.. It’s an interesting fact that in very high stakes national security debates and in many other types of high stakes policy debates as well, people don’t think it’s possible to make very granular probability estimates. Sometimes they seem to act as though “things are going to happen,” and there’s “maybe” and “things aren’t going to happen.” Sometimes they act as though there will be only three levels of uncertainly. Sometimes they might act as if there are five or seven.

.. It’s fair to say that the vast majority of college-educated people believe that probability theory is useful in estimating the likelihood of a fair coin landing heads five times in a row. They think probability theory is useful if you’re playing poker and you’re drawing cards from a well-defined sampling universe. These are classic domains for frequent disk statistics. The question we’re confronting here is, what are the limits on the usefulness of probability? To what extent is it useful to elicit probability judgment for seemingly unique historical events? On this page, I list a number of situations in which people find it vexing to make probability judgments. The first one is, is there intelligent life elsewhere in the Milky Way?

.. One of his particularly insightful columns was one he wrote in late 2002 before the 2003 invasion of Iraq. He posed the question, is Iraq the way it is because Saddam is the way he is, or is Saddam the way he is because Iraq is the way it is?

..  If you make the mistake of missing something, missing a threat, make sure you don’t miss another one. And if you make the mistake of having a false positive on a threat, make sure you don’t make another false positive right away. Show, at least, that you’re responsive to the political blame game calculus.

.. It’s even possible to take a policy like, say, the invasion of Iraq, which almost everybody has bailed on, but you could construct a counterfactual that says, “Well, you know what? If you think things are bad now, you have no idea how bad they would have been if Saddam had stayed in power.” There were people who defended the Vietnam War or the Iraq War on those grounds even after most opinion had bailed out. But counterfactuals are a very interesting and integral part.

.. These narrative accounts that we give are not only always wrong, they’re always misleading because they’re so comforting and persuasive. We want to tell stories, we like to hear stories, and once we’ve heard the story, we now understand the situation and we’re done. It’s pathological, our love of stories in this sense of trying to be accurate about things that might happen, or trying to understand the levels of what’s going to. In biology, we encounter this all the time. There are so many simplistic stories relating to evolution, and you look a little deeper and it just isn’t that simple ever.

.. If you look at the failure of the intelligence community in the case of WMD, it was a failure of storytelling, not so much a failure of evidence interpretation. Retrospectively, you could tell a story of Saddam deliberately pretending like he had weapons of mass destruction, there’s a story about his generals lying to him; none of those stories were told beforehand.

Had those stories been told beforehand, the evidence could have been interpreted differently. The same evidence could have been interpreted differently.

.. I think Gordon Moore’s law, which, if you read the original paper, was based on five data points, one of which was null and four were just over a two-year period. He wrote this curve, he extrapolated it for ten years, and then the industry said, “Ah, that’s what’s going to happen,” and everyone worked that straight line for the next thirty or forty years. It completely transformed their world. It was exactly an example of a prediction changing the outcome, because without that I don’t think that Intel and all the other chipmakers would have known what to aim for. That told them what to aim for.

.. Foxes know many things but a hedgehog knows one big thing. The hedgehogs come in many flavors in expert political judgment. There are free market hedgehogs, there are socialist hedgehogs, there are boomster hedgehogs, there are doomster hedgehogs. They come in a variety of ideological complexions. This particular hedgehog was an ethnonationalist hedgehog.

.. Or Paul Krugman recently being interviewed by Fareed Zakaria about the recommendation he made to the Greek population about they should vote no on the loan referendum. Zakaria was asking him whether he had made a mistake, and Krugman said, “Yes, I think I did make a mistake. I think I overestimated the competence of the Greek government,” meaning he didn’t think they would be so dumb as not to have a backup plan if the European leaders didn’t move

.. And of course, academia encourages specialization.