With the financial crisis and the blame being heaped onto models I wonder if there was any future for models or what future models might look like.
One possiblity might be the advance of agent based modeling which has not had made much headway against DSGE models or even heterogenous agent DSGE models. Leigh Testfatsion has been a contributor to this field for a long time. Tyler Cowen maligns it:
What's the important innovation behind intelligent agent modeling? To introduce lots of arbitrary assumptions about behavior? Greater realism? Complexity? Considerations of computability? Learning? We already have enough "existence theorems" as to what is possible in models, namely just about everything. The CGE models already have the problem of oversensitivity to the initial assumptions; in part they work because we use our intuition to calibrate the parameters and to throw out implausible results. We're going to have to do the same with the intelligent agent models and the fact that those models "sound more real" is not actually a significant benefit.
What can be done will be done and so people will build intelligent models for at least the next twenty years. But it's hard for me to see them changing anyone's mind about any major outstanding issue in economics. What comes out will be a function of what goes in. In contrast, regressions and simple models have in many cases changed people's minds.
But Alex Tabarrok is more optimistic:
I see bringing experimental economics and I-A modeling closer as an important goal with potentially very large payoffs. Here, for example, is my model for a ground-breaking paper.
1) Experiment
2) I-A replication of experiment (parameterization)
3) I-A simulation under new conditions
4) Experiment under the same conditions as 3 demonstrating accuracy of simulation
5) I-A simulation under conditions that cannot be tested using experiments.
I am also more optimistic since reading about swarm models. (See an old post.) I'd complement Alex's approach with the advent of greater amounts of data that is becoming more available. For instance, the following claims are made via Andrew Gelman:
1. More data beats better algorithms (Some agreement.)
2. The End of Theory: The Data Deluge Makes the Scientific Method Obsolete (Dissent and agreement within link.)
3. Some convergence in using priors (intuition), large databases, visualization and modeling language.
Friday, January 16, 2009
Thursday, January 15, 2009
Model builders and model users
First, John Quiggin poses the question: Bad models or Bad Modelers? If an underlying assumption of the model is bad is the model at fault or is the modeler at fault? After all models don't make assumptions, people do. Why should we accept what the models tell us without close examination? Every year compter rankings of college football teams are generated and these rankings are constantly being disputed. There is a healthy disrespect for models in college football that does not seem to carry over to finance and economic models.
Second, from Joe Nocera (NYT):
There were the investors who saw the VaR numbers in the annual reports but didn’t pay them the least bit of attention. There were the regulators who slept soundly in the knowledge that, thanks to VaR, they had the whole risk thing under control. There were the boards who heard a VaR number once or twice a year and thought it sounded good. There were chief executives like O’Neal and Prince. There was everyone, really, who, over time, forgot that the VaR number was only meant to describe what happened 99 percent of the time. That $50 million wasn’t just the most you could lose 99 percent of the time. It was the least you could lose 1 percent of the time. In the bubble, with easy profits being made and risk having been transformed into mathematical conceit, the real meaning of risk had been forgotten. Instead of scrutinizing VaR for signs of impending trouble, they took comfort in a number and doubled down, putting more money at risk in the expectation of bigger gains. “It has to do with the human condition,” said one former risk manager. “People like to have one number they can believe in.”
(see a critique of the article here)
There is some truth to the notion that once an unsophisticated user has been exposed to a concept long enough this notion can take become the TRUTH and in a sense this is what has happened to VAR. The more sophisticated users/modelers who understand the assumptions behind models will also be lulled into complacency if the novice user e.g. CEOs don't take the trouble to understand the models and act as though there were no limitations to the models. If my boss is not worried then why should I worry?
The incentive to worry about the 1 percent is also not present. Why devote resources to the small probability of a catastrophe when everyone else isn't doing it? After all, ‘a sound banker, alas, is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way with his fellows, so that no-one can really blame him.’ (Keynes)
As Nocera points out in the article many don't believe that VAR models are useless but there is an element of human judgement that needs to be used every time the numbers are scrutinized. So, should all risk models come with a warning e.g. "This model will only behave as it has been programmed to behave. Use at your own risk".
Unfortunately, disclaimers such as these are ubiquitous - almost like end user license agreements when software is installed - that I almost never read anything like Terms and Conditions or Disclaimer any more.
There is a human element to all financial crisis and it is neither stupidity nor ignorance. It is greed.
Second, from Joe Nocera (NYT):
There were the investors who saw the VaR numbers in the annual reports but didn’t pay them the least bit of attention. There were the regulators who slept soundly in the knowledge that, thanks to VaR, they had the whole risk thing under control. There were the boards who heard a VaR number once or twice a year and thought it sounded good. There were chief executives like O’Neal and Prince. There was everyone, really, who, over time, forgot that the VaR number was only meant to describe what happened 99 percent of the time. That $50 million wasn’t just the most you could lose 99 percent of the time. It was the least you could lose 1 percent of the time. In the bubble, with easy profits being made and risk having been transformed into mathematical conceit, the real meaning of risk had been forgotten. Instead of scrutinizing VaR for signs of impending trouble, they took comfort in a number and doubled down, putting more money at risk in the expectation of bigger gains. “It has to do with the human condition,” said one former risk manager. “People like to have one number they can believe in.”
(see a critique of the article here)
There is some truth to the notion that once an unsophisticated user has been exposed to a concept long enough this notion can take become the TRUTH and in a sense this is what has happened to VAR. The more sophisticated users/modelers who understand the assumptions behind models will also be lulled into complacency if the novice user e.g. CEOs don't take the trouble to understand the models and act as though there were no limitations to the models. If my boss is not worried then why should I worry?
The incentive to worry about the 1 percent is also not present. Why devote resources to the small probability of a catastrophe when everyone else isn't doing it? After all, ‘a sound banker, alas, is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way with his fellows, so that no-one can really blame him.’ (Keynes)
As Nocera points out in the article many don't believe that VAR models are useless but there is an element of human judgement that needs to be used every time the numbers are scrutinized. So, should all risk models come with a warning e.g. "This model will only behave as it has been programmed to behave. Use at your own risk".
Unfortunately, disclaimers such as these are ubiquitous - almost like end user license agreements when software is installed - that I almost never read anything like Terms and Conditions or Disclaimer any more.
There is a human element to all financial crisis and it is neither stupidity nor ignorance. It is greed.
Reading Jennifer McMahon
Both Promise Not To Tell and Island of Lost Girls were good. They were both page turners although the second was a little more predictable in terms of ending. Both books are written so that the past and present are interleaved into the book so the reader switches from the present to the past. Fortunately they both advance the story so it was not disconcerting for me. I enjoyed both books but I'm not sure if I'd read another one of her future books. The style and approach is getting a little stale as well as the type of stories that she tells. She is quite a good writer though so I think she might be able to pull of another one of these genre very well, so who knows.
Wednesday, January 14, 2009
American Shaolin
Read Matthew Polly's American Shaolin - the time he spent in China learning martial arts at the Shaolin Temple. It was a good read and I liked it better than some of the travel articles that he had written on Slate. It was good to have the chapters organized under his experiences (sometimes compared to his expectations as an American) rather than in chronological order.
It brought back some memories of growing up:
1. Wong Fei Hung shows on TV played by Kwan Tak Hing. See also here. The most recent version was I saw was Jet Li in Once Upon A Time In China.
2. The book also mentions Jet Li's first movie Shaolin Temple and how the movie craze then swept the Wushu world.
3. The One Armed Swordsman played by Ti Lung.
4. The many Police Story movies by Jackie Chan and Sammo Hung
It brought back some memories of growing up:
1. Wong Fei Hung shows on TV played by Kwan Tak Hing. See also here. The most recent version was I saw was Jet Li in Once Upon A Time In China.
2. The book also mentions Jet Li's first movie Shaolin Temple and how the movie craze then swept the Wushu world.
3. The One Armed Swordsman played by Ti Lung.
4. The many Police Story movies by Jackie Chan and Sammo Hung
R wish list
Was part of an R discussion - I haven't really used R except for the VARS package and I've been trying to make the switch from SAS but it hasn't worked because of all the pre-processing that I have to do to get a data set for analysis.
My wish list for R is as follows (and they may already be there just not to my mediocre knowledge or quick Google searches of the R discussion list):
1. An input statement for processing text files like SAS - this is key to reading public use files that are usually very large and having to avoid reading the entire file using read.table or the fortran syntax for reading files.
2. Several commenters noted that you can read files without using data frames and I was not able to find a reference to it on the R-discussion list. I'm thinking that this is achieved using vectors or matrices but haven't quite figured it out yet.
3. A first dot and last dot syntax similar to SAS or an egen statement similar to STATA.
4. it would be nice if the R foreign package has a keep or drop statement so that I don't have to read the entire data set into memory. I tried to read the public use version of World Values Survey Data which was in Stata xpt format but the memory limitations on my computer couldn't handle it.
I realize that R is NOT a data processing package and something like Perl could also work BUT it's always nice to have everything integrated instead of having to deal with two languages and porting abck and forth between languages to do what I consider basic data processing tasks. I consider data analysis 90 percent processing and 10 percent analysis and then another 100 percent fooling around with different packages to get the results I want.
My wish list for R is as follows (and they may already be there just not to my mediocre knowledge or quick Google searches of the R discussion list):
1. An input statement for processing text files like SAS - this is key to reading public use files that are usually very large and having to avoid reading the entire file using read.table or the fortran syntax for reading files.
2. Several commenters noted that you can read files without using data frames and I was not able to find a reference to it on the R-discussion list. I'm thinking that this is achieved using vectors or matrices but haven't quite figured it out yet.
3. A first dot and last dot syntax similar to SAS or an egen statement similar to STATA.
4. it would be nice if the R foreign package has a keep or drop statement so that I don't have to read the entire data set into memory. I tried to read the public use version of World Values Survey Data which was in Stata xpt format but the memory limitations on my computer couldn't handle it.
I realize that R is NOT a data processing package and something like Perl could also work BUT it's always nice to have everything integrated instead of having to deal with two languages and porting abck and forth between languages to do what I consider basic data processing tasks. I consider data analysis 90 percent processing and 10 percent analysis and then another 100 percent fooling around with different packages to get the results I want.
Thoughts on randomization
Mostly triggered by Chris Blattman's advice to PhDs:
"The randomized evaluation is just one tool in the knowledge toolbox. It's currently the rage, but that means it will probably be old news by the time you finish your PhD."
One of the problems with randomized trials is that is is a black box. We understand very little or we may think we understand a lot. There is also a lot of potential subgroup interaction that needs to be tested.
All this points to the fact that if we have to do a randomized trial then we don't really understand the mechanism of how the treatment works (and this also applies to medical "science"/drug therapy, etc.). And if it does work to our expectations then it validates our priors and perhaps advances the field a little. But does it really advance our understanding of the causal underlying mechanism? All we can point to are suggestions that our limited understanding is validated but we could still be spectacularly wrong.
Another problem with randomized trials is that it usually is never the last word. (Perhaps repeated randomized trials can provide the last word, but rarely one randomized trial.) Again, this is because if a theory accords with my priors and the results of my hypotheses are rejected it doesn't seem to lower my priors as much as it should - mainly because the "theory" sounds so sensible and plausible. So it must be something with the way the trial is conducted. For instance, the effects of Head Start on children and the disappointing First Year results - yet the underlying premise of Head Start is so strong that it will not go away.
Randomized trials also do not address the question: How will it work for me? And this is particularly true for drugs. I really do not care about average treatment effects of the average treatment effects on my subgroup. And it is this thinking that leads to experimentation and continuing treatment using "less than acceptable" methods or alternative methods. This would be the test of our understanding - if we can predict individual results then we can claim to have the final word on causality.
"The randomized evaluation is just one tool in the knowledge toolbox. It's currently the rage, but that means it will probably be old news by the time you finish your PhD."
One of the problems with randomized trials is that is is a black box. We understand very little or we may think we understand a lot. There is also a lot of potential subgroup interaction that needs to be tested.
All this points to the fact that if we have to do a randomized trial then we don't really understand the mechanism of how the treatment works (and this also applies to medical "science"/drug therapy, etc.). And if it does work to our expectations then it validates our priors and perhaps advances the field a little. But does it really advance our understanding of the causal underlying mechanism? All we can point to are suggestions that our limited understanding is validated but we could still be spectacularly wrong.
Another problem with randomized trials is that it usually is never the last word. (Perhaps repeated randomized trials can provide the last word, but rarely one randomized trial.) Again, this is because if a theory accords with my priors and the results of my hypotheses are rejected it doesn't seem to lower my priors as much as it should - mainly because the "theory" sounds so sensible and plausible. So it must be something with the way the trial is conducted. For instance, the effects of Head Start on children and the disappointing First Year results - yet the underlying premise of Head Start is so strong that it will not go away.
Randomized trials also do not address the question: How will it work for me? And this is particularly true for drugs. I really do not care about average treatment effects of the average treatment effects on my subgroup. And it is this thinking that leads to experimentation and continuing treatment using "less than acceptable" methods or alternative methods. This would be the test of our understanding - if we can predict individual results then we can claim to have the final word on causality.
Tuesday, January 13, 2009
Dilemma of an economist
This is from Ariel Rubinstein's Dilemma of An Economic Theorist but I think it applies to economists as a whole:
"What on earth am I doing? What are we trying to accomplish as economic theorists? We essentially play with toys called models. We have the luxury of remaining children over the course of our entire professional lives and we are even well paid for it. We get to call ourselves economists and the public naively thinks that we are improving the economy’s performance, increasing the rate of growth, or preventing economic catastrophes. Of course, we can justify this image by repeating some of the same fancy sounding slogans we use in our grant proposals, but do we ourselves believe in those slogans?"
By the way, Ariel Rubinstein's website is like entering what I might have thought to be a run of the mill book store but discovered it to be an amazing place for browsing. Check out his Cafe Poster.
"What on earth am I doing? What are we trying to accomplish as economic theorists? We essentially play with toys called models. We have the luxury of remaining children over the course of our entire professional lives and we are even well paid for it. We get to call ourselves economists and the public naively thinks that we are improving the economy’s performance, increasing the rate of growth, or preventing economic catastrophes. Of course, we can justify this image by repeating some of the same fancy sounding slogans we use in our grant proposals, but do we ourselves believe in those slogans?"
By the way, Ariel Rubinstein's website is like entering what I might have thought to be a run of the mill book store but discovered it to be an amazing place for browsing. Check out his Cafe Poster.
Subscribe to:
Posts (Atom)