On forecasting (yet again)

Rajiv Sethi points out much more eloquently than I can the complications with forecasting or why we did not predict the financial crisis. (He uses prediction markets as an example.)

... the very same market characteristics that serve to enhance predictive accuracy in the case of exogenous events could undermine accuracy in forecasting endogenous events. Accurate forecasting of exogneous events requires broad participation and high levels of market visibility and liquidity, so that decentralized information can be effectively aggregated. But in the case of endogenous events, the more reliable a market is perceived by the public to be, the greater the incentives to manipulate prices at the margin. The problem is especially severe when there is a positive feedback loop between subjective beliefs and objective probabilities, ...

Thus, the more likely the event is going to occur, the more skewed my incentive would be to make it occur (if it rewards me). If everyone were like me (a representative agent) then it would likely become self-fulfilling. If there are less of me but more of the opposite then the outcome would not occur therefore making the prediction and the forecast invalid. Since very few people had an interest in seeing the crisis occur the forecast that would be one is then invalidated.

This seems to represent a paradox in forecasting or why as some have claimed:

... economists did something even better than predict the crisis. We correctly predicted that we would not be able to predict it. The most important part of the much-maligned Efficient Markets Hypothesis (EMH) is that nobody can systematically beat the stock market. Which implies nobody can predict a market crash, because if you could, then you would obviously beat the market.

The above argument does not rely on EMH but then again, I don't really believe that the crisis was not forecasteable because people were consciously trying to not make it so. This is why I think that the gleeful economist who proclaims that crises are unpredictable and that economists even predict that they cannot predict it is like a fool who says he is infinitely wise because he knows that he knows nothing.

However, I believe that there is a feedback in financial markets forecasting and behavior of actors in the markets that make forecasts inaccurate.