What the Oregon health study shows or doesn't show

Some thoughts about the Oregon Health Study - which is being dissed by MR here and here.

First off the comparison of this study to Reinhart and Rogoff is way off base. Reinhart and Rogoff were recalcitrant in sharing their data. Had they been more forthcoming in the first place it is possible that someone would have pointed out the Excel error much earlier. Instead they left all the researchers puzzled over their attempts to replicate their findings. This is NOTHING like Reinhart Rogoff.

What if they had found positive effects? Then I think the debate would have shifted to size effects and whether the results were meaningful in any sense. The critics are leaping to the "no result" finding as a way to justify their opposition. They would probably have leaped to a positive result by pointing out the lack of meaningful results regardless.

RCTs are pretty useless in this debate. Think of their outcomes - blood pressure, hypertension and cholesterol levels. How can having insurance by itself promote better outcomes? To actually change these outcomes, insurance holders have to actually change their behavior. There is a lot of behavioral economics evidence out there to argue against anyone actually doing anything about this regardless of health insurance status.

This is an example of RCTs taking away the analysts' ability to think straight. Retrospection would have predicted the null effect. The low income population makes them even more constrained in their ability to affect lifestyle changes that would positively affected the outcome measures. I have some doubt as to whether even a high income population would have been able to affect the lifestyle changes necessary to have a detectable effect.

This is an example of RCT measuring the wrong outcome. We only measure what we can see or even worse - what is easiest to measure. By some kind of fluke these are the outcomes that are hardest to change. I have to change my diet, my exercise regimen, my sleep habits, and so forth. If this is hard for a middle class person how much harder would it be for a low income group?

Effects of the recession on health? Is it possible that the stress of the recession and the job market is driving the results in both treatment and control?

This randomized study should have been titled something like "How much do people take their doctors' advice in changing their lifestyles? An analysis of low income population using Medicaid lotteries" or something similar. Would the findings have made any headlines then?

Alternatively, if the study participants were given medications to treat their hypertension, cholesterol, etc. then the study would have been about the efficacy of drugs on the low income population. Or perhaps prescription drug compliance among the low income population.

The comeback "You still buy health insurance don't you?" is a reasonable response contrary to what Tyler Cowen thinks. And this returns to the possibility that the benefits of ACA and expansion of Medicaid was oversold. Why do we buy health insurance - surely not to become healthier. We buy insurance to preserve the status quo, i.e. I don't want my health to get worse and if it gets worse then at least I can do something about it. I have health insurance and I can go to the doctor. And when we think about health we generally don't think in terms of cholesterol or blood pressure. We think of actual illnesses, pain, discomfort, etc. and again evidence from behavioral economics may be useful here.

What is going here I think is that possibly and perhaps in fact, very likely that supporters of ACA felt they had to justify the expansion of Medicaid by some evidence based reason and like their opponents who will grab at anything they can to fight against it they fell onto the belief that health insurance improves overall health - not specific illnesses, but overall health because it was easier to measure in an RCT.

Instead of making arguments for ACA on grounds of equity and perhaps even a right to adequate care and a right not to impose the costs of emergency room visits on others, etc. they have backed themselves into a corner with this null finding.

Does correcting for self selection change the policy question

Consider an experiment of whether job training after layoff increases the probability of being re-employed. A ‘naive’ treatment effect would be to compare the effects of those who enrolled in job training and those who didn’t. The estimated effect would then be the difference in likelihood of being employed for those with job training and those without. The policy question addressed here is whether job training increases the likelihood of employment.

But the econometrician would argue that those who did not enrol in job training are different from those who did and that these characteristics are unobservable to him (the econometrician, e.g. motivation might be unobservable). In order to accurately estimate the impact of job training one would have to compare apples to apples, i.e. those who applied for job training but were (randomly) rationed out of the program. This gives the correct estimated impact. But the policy question now seems to be whether those who applied for job training but were not denied increases the likelihood of being employed. I would argue that this is NOT the policy question of interest.

The policy instrument is to shift people into job training - assuming that the impact is or can be positive. But by estimating the impact only for “motivated” people this naturally assumes that the unmotivated will not be treated. Suppose the following:

• A randomized control trial of a job training program is run and impacts estimated.
• The impacts are found to be large and cost benefit analysis shows that the benefits are positive on net.
• What happens when the program is scaled up, i.e. rolled out to the entire population of unemployed (instead of just to the treatment and control who were "similar" in the trial)? Should we assume that the impacts would still be the same as in the RCT? Are the participants on the now scaled up program still similar? An RCT advocate would argue yes - but - isn't the original intent of scaling up a program to get as many people as possible to participate regardless of the original composition of the treatment and control groups?
• Should the scaled up program be the same as the RCT, i.e. a static program that doesn't enroll anyone but just allows the "motivated" to enroll themselves? What if there was an effort to try to get the recalcitrant unemployed into the program - after all since the benefits are positive, don't we want to extend the benefits to as many as possible? If there were such an effort would the estimated impacts still be the same as in the trial?
• Suppose that after the completion of the trial we find that the population of unemployed has changed so that there are now more women than men? Do we deny one gender the treatment because it is no longer the same as those in the randomized trial?

Why we look for a single cause

In a previous post, I whined about the state of econometrics, particularly the obsession with instrumental variables and single factor causes. The obvious question is why does this state of affairs persist?

"I only wish we had a single agent causing all the declines," Pettis says. "That would make our work much easier."

This is from National Geographic on colony collapse disorder.

When CCD first hit, many people, from agronomists to the public, assumed that our slathering of chemicals on agricultural fields was to blame. Indeed, says Jeff Pettis of the USDA Bee Research Laboratory, "we do find more disease in bees that have been exposed to pesticides, even at low levels." But CCD likely involves multiple stressors. Poor nutrition and chemical exposure, for instance, might pummel a bee's immunities before a virus finishes the insect off.

It's hard to tease apart factors and outcomes, Pettis says. New studies reveal that fungicides—not previously thought toxic to bees—can interfere with microbes that break down pollen in the insects' guts, affecting nutrient absorption and thus long-term health and longevity. Some findings pointed to viral and fungal pathogens working together.

Econometric Warriors of Truth

One of the most interesting aspects of Jared Diamond’s work is the fact that there has been no attempt at finding THE truth to THE question of economic growth or collapse. He freely admits that he will not do that and points to multiple sources of causality.

It is unclear to me why economists and econometricians persist on attempting to find THE ultimate cause of some question such as financial crises. Part of the reason they have embraced randomized trials is because they divine that they can finally grasp the answer in their hands instead of exploring the deeper questions of multiple causes or intermediate outcomes that may affect the final outcome of interest.

What is even more surprising from my naive point of view is that while they have become obsessed with exogeneity and strength of their instruments they have moved away from goodness of fit statistics and analysis of variance. What does an econometric model really say when an instrument is strongly exogenous with a large t-statistic but the fit of the model is low? Even more so, what if the variance explained by the instrument is even lower? And to sound even more radical, I am surprised that few have adopted MIMIC models. Is this perhaps because (gasp) it smacks too much of structural equations without microfoundations?

In their pursuit for their idealized version of truth, economists and econometricians have perhaps more than ever acquired an extreme form of tunnel vision are unable to see the forest for the trees.

Guns and crime analysis rethought

After posting on Lott’s analysis I’ve come to the conclusion that without accounting for trends in crime the estimated effects may in fact be biased. Like Lott, I don’t believe that including quadratic or cubic terms is the way to go either. The charts in his book “More Guns Less Crime” practically scream out ‘event study’ but this was not the approach used perhaps because event studies seem mainly limited to finance.

Briefly as I vaguely remember it, we are interested in how stock prices of a firm respond to an announcement where the date of the announcement is known. In order to isolate the effects of the announcement on the firm the stock price is corrected using time series regression to remove industry or overall stock price effects.

It is unclear to me at this point how an event study can be implemented in Lott’s analysis but I think that the overall trend in crime needs to be taken into account. Since the analysis is at either the state or at the county level and the passage of the law/date of concealed carry law is at a state level, the county/state level crime rates (i.e. robbery, murder, etc.) need to be isolated from the overall trends. The only way to do this perhaps is to regress the state/county level crime rates on the national (or regional) crime rates (and it is unclear whether it is essential to match types of crime at the state/county level to its corresponding type at the national level) and extract the residuals from the regressions.

The residuals would then be used as the dependent variable in the analysis that are in Lott’s book.

Returns to education

I’ve been going over the econometrics literature on the returns to education (see for instance, Card) and it’s the first time I’ve looked at this and I’m a little surprised at how it’s presented. The estimated return is assumed to be linear in years of education and as a baseline method (first approximation) this works well although this is not how I would think of as a return to education. Unfortunately, the literature seems to be fixated on this method.

My natural inclination is to think of returns to a high school degree versus high school dropout or to measure directly the returns to a college degree. In terms of the latter, it would seem to be the wage premium of a BA over a HS but this premium when averaged over the population confounds years of experience, ability and time/year effects so that it becomes hard to separate.

If I’m about to invest $200,000 in a college degree I would like to know the return to this investment, not the return to an additional year’s of education. If I were to think of this as an investment project, I would I would first calculate the discounted PV of the lifetime stream of income that the completion of a college degree would confer versus say, the discounted PV of a HS graduate. Supposing that the PV of a college graduate is $2 million versus $1 million for a HS degree, the the NPV would be $1 million. Thus, the return would be ($1 million-$200,000)/$200,000 or 400%.

But the calculation of any streams of income is a road that is fraught with perils. It makes a lot of assumptions about baseline earnings and earnings growth (as well as inflation rates and interest rates) and in some cases, small differences can matter. If for some reason the PV of a college graduate is only $1.5 million, then the return falls to ($500,000-$200,000)/$200,000 or 50%. If the PV is $1.2 million for a college graduate then the NPV is 0 ($1.2 million - $1 million - $200,000) or a return of 0.

The lifetime incomes of $2 million, $1.5 million, and $1.2 million do not seem to be a large difference yet the return has fallen from 400 percent to 0 very quickly. Now, some would say that the PV of the lifetime income of HS graduate should not be deducted as a cost to calculate NPV and I would say perhaps but I need a way of comparing alternative investments and not getting a college degree represents an opportunity cost to getting a college.

What I've always wondered about convergence

But was afraid to ask until it was asked for me: Why does anyone care about the distinction between convergence in probability and almost sure convergence?

Some answers:
1. "Suppose a person takes a bow and starts shooting arrows at a target. Let Xn be his score in n-th shot. Initially he will be very likely to score zeros, but as the time goes and his archery skill increases, he will become more and more likely to hit the bullseye and score 10 points. After the years of practice the probability that he hit anything but 10 will be getting increasingly smaller and smaller. Thus, the sequence Xn converges in probability to X = 10.Note that Xn does not converge almost surely however. No matter how professional the archer becomes, there will always be a small probability of making an error. Thus the sequence {Xn} will never turn stationary: there will always be non-perfect scores in it, even if they are becoming increasingly less frequent."
Also, almost sure convergence implies convergence in probability.

2. The most useful intuitive understanding I've been taught is that almost sure convergence guarantees that X_n be far from X (ie. further than any epsilon) only a finite number of times. Convergence in probability leaves open the possibility that X_n will be far from X an infinite number of times.
The best example I have to illustrate that is if you take Y_n as a Bernoulli(1/n) random variable. Clearly Y_n converges to 0 in probability, but it doesn't converge almost surely. Y_n will always be 1 for an infinite number of n's. You can see this from the second Borel-Cantelli Lemma.
Of course, I've got no idea if the distinction has any practical relevance for econometrics.

3. Convergence in probability is a form of weak convergence. Your students should understand the difference between convergence and weak convergence -- the difference is huge. If you have a sequence x_n, then weak convergence means that f(x_n) --> L for some f. This does not mean that x_n converges, but only that some attribute converges.
For example, you can ask, given N asset prices, if the sum of these prices converges to 1, does that mean that each individual asset price converges to something? No. Here f is the operation of taking the sum. It could be average, variance, integration against a test function, the infimum of a large set of integrations against test functions, whatever. ... Weak convergence, point-wise convergence, and uniform convergence are different concepts and useful ideas to understand, and they appear over and over again in different forms whatever branch of math you are studying.

Daylight savings time or when should you trust economics papers

I was a little irked to see this again especially in the context of Felix Salmon's post on When Can You Trust Economics Papers? My response would be never.

This is a reprise of an earlier New Economist post and my own comments. From New Economist:

In my naivety I had assumed most social scientists understood how to model interaction effects. Then I read this post by Omar on orgtheory.net, and realised maybe I was wrong:

So we are agreed interaction models are awesome. However, as your stats 101 teacher told you, you have to be careful about two things: (1) never omit the main effects. Thus you don’t test hypothesis 1 using any of these specifications:

Y=a+b1X+b2XZ+e
Y=a+b1Z+b2XZ+e

or Alanis forbid:

Y=a+b1XZ+e

And (2) when interpreting b1 and b2 in the fully specified model, remember that those effects are conditional on the value of the other variables. b1 is now the effect of X on Y when Z=0 and b2 is now the effect of Z on Y when X=0. If your variables don’t have a meaningful zero point (like a racial attitudes scale), center them at their mean so that you can say “b2 is the effect of being Southern on voting republican for those who have average levels of racial animus towards blacks.”

Seems simple. Everybody knows this. Why am I even explaining this to you? Well, as noted by Brambor, Clark and Golder (2006) in a recent article in Political Analysis, a survey of 156 articles published in the major Political Science journals shows that only 10% of researchers specified their interaction models correctly. A large chunk of them outright omitted main effects, which can lead to incorrect significance tests of the interaction term. In some of these articles the entire contribution was riding on the interaction term. So things are not so simple. Consider the horror:

In an award-winning article in the American Political Science Review, Boix (1999) examines the factors that determine electoral system choice in advanced democracies. He makes two main conclusions. First, ethnic or religious fragmentation encourages the adoption of proportional representation in small and medium-sized countries (621). He draws this conclusion based on a model that includes an interaction term between ethnoreligious fragmentation and country size. However, he does not include either of the constitutive terms. When these terms are included, there is no longer any evidence that ethno-religious fragmentation ever affects the adoption of proportional representation (italics added).

You should read the article to see other horror stories. The lesson: if your dissertation/paper is riding on an interaction effect, don’t be a fool. Estimate a fully specified model.

It looks as though the Daylight Savings Time paper by Grant and Kotchen did not do this (or at the very least, if they did, it is not at all obvious that they did). See their Equation 2, Table 4 and 5. The treatment variable needs to be included as a main effect which they did not. Their conclusions rest entirely on the interaction of the treatment with year.

How to treat outliers

I came across this post on outliers and was surprised to read that it pointed back to Mark Thoma who in his zeal to debunk the Laffer Curve advocated throwing out an outlier. I agree with Crooked Timber that outliers need to be treated with caution and should be excluded only after careful consideration, not because it doesn't accord with the results that we would like to have. Note that this does not say that I agree with the existence of the Laffer Curve as the WSJ was in quick to publish.

Here is how we've looked at outliers at where we work:
1. Outliers are usually but not always indicative of some possible data entry error. So these are excluded only if after checking that it was an error and there is no recoverable data, it is then set to missing.
2. If the variable is set to missing and is part of a set of predictor/covariate/independent variable (I never know which terminology to use because where I am each discipline uses her own terminology) then some statisticians might advocate some kind of imputation. (I'm not a big fan of imputation but I'll go with it for now.)
3. If outliers are valid observations then they are part of the empirics that need to be modeled, explained, what have you. We don't just throw it out because it is incovenient and does not fit with our idea of the world.

As an example, one of the problems we had was something like this:
Q. How many times did you (the parent) spank your child in the past week?
R. It must have been over 100 times.
(It was duly coded as 100.)

What was the best way to handle this record? It was obviously an outlier. In the end, after some hand wringing we set it to the maximum (the maximum excluding this outlier, that is). Was it better to set it to missing? I don't know. No imputation was performed on this variable because it was an outcome - the study wanted to test the treatment effects of some program on parental practices/style.

What influences inflation expectations?

This post made me wonder:

" ... according to Professor Eric Johnson, may be the frequency with which consumers are seeing higher prices. “Things that you buy more frequently and that have large percentage increases will weigh more in people’s perception of inflation,” Johnson was quoted as saying.
He elaborated in the article with the following example: a person paying an extra $25 to fill up the gas tank is reminded of that cost once a week, or more often if you count the times he or she sees a $4-per-gallon price in giant numbers on a sign. In contrast, a rent increase of $100 would only happen once a month but would have the same financial impact."

It seems like this should be easy to test:
Volatile components of the CPI should have a larger effect on inflation expectations.
The data for CPI consists of food, energy and all other items less food and energy (the assumed less volatile components).
Food and/or energy should have larger "impacts" on inflation expectations.
In a regression of inflation expectations on change in food, energy and all other prices, the coefficients of food/energy is expected to be larger than the coefficient on changes of all other prices. (I think this is correct. I realize that the size of the coefficient does not always imply that the effect is larger but in this case it should work because all the right hand side variables are measured in the same units - percent change.)
To be more precise, it is not the size of the coefficient but the average effect as measured by the coefficient multiplied by the average changes in food, energy and all other prices that gives the size of change.

At the risk of further embarassing myself since I haven't engaged in any real econometrics in years here are some results:
I use non-seasonally adjusted data on CPI from FRED and inflation expectation from Michigan's Survey of Consumers.

The following plots the mean and median expectations of percent changes in inflation with the percent change in food.

This is a plot of the same expectations with percent change in energy:


And the following is with percent change in all other prices:

Inflation expectations are of several orders of magnitude larger than actual percent changes in prices. Surprisingly changes in all other goods excluding food and energy are more variable than changes in food prices (mean percent change is 0.34 versus 0.31).

What about the results from an estimated regression equation? The mean percent change in each of the price indices multiplied by its regression coefficient is summarized below - the estimate is the middle line while the 95% confidence intervals are the top and bottom of the bars:


Surprisingly, (if my naive estimates are correct), changes in food and energy prices have smaller effects on consumer expectations of inflation than changes in all other prices.

I am also reminded by Jim Hamilton that time varying volatility might be present in the dependent variable. (In fact, it looks like it's present in all the series but I don't know how to handle this.) I reestimated the above assuming a GARCH(1,1) and find that the results do not change substantially. (I haven't plotted the coefficients yet and maybe at some point I will update this post with the plots.)

Can we determine causality without really determining causality?

The recent rise in the price of oil ($139 today) has been determined to be the cause of the following:
1. The fall in the number of miles driven
2. The fall in consumption of gasoline
3. The fall in the sales of light trucks
See a summary here by Jim Hamilton. Yet how has causality been determined? Certainly, not by randomized trial or by any econometric methods - except for ocular regression. Two time series are running in different directions therefore it has to be the cause. Fifty years from now, when we look at these time series again can we actually conclude that the price of oil cause the declines?

Interaction variables yet again

This is the first time I've put any thoughts on this but I had thought that everyone had read the post on New Economist and the links in the post. The catalyst for this post was a paper by Matthew Kotchen and Laura Grant, "Does Daylight Saving Time Save Energy? Evidence from a Natural Experiment in Indiana". What bothered me a little was that they labeled their paper "Very Preliminary" yet allowed themselves to present their findings in the Wall Street Journal. See Marginal Revolution's post for the link. What bothered me even more was that this paper was presented at the NBER and no one seemed to have caught the error. This indicates to me that economists make this error more often than not -- just like political scientists. See this link for the same error made by political scientists.

However, I have to admit that even after getting a PhD I was not pointed to this type of error until I started working - by statisticians. One analyst (another PhD economist) proposed estimating an equation with interaction terms but without including all of the variables as main (or level) variables. The statistician on the project had to point out that this is not correct. I can now see why this is the case. For instance, let's say we want to estimate:

Y = a + bX1 + cX2 + dX1X2

1. From an ANOVA standpoint there is no reason to exclude X1 and X2 separately (one or both) and just include X1X2.
2. Leaving out one of the main effects (or level variables), for instance, X2 is tantamount to assuming/imposing the restriction c = 0. There is no a priori reason to do this. Econometrics lets us test this restriction and there really is no harm to keeping it in.
3. Leaving out one variable is similar to doing model selection by dropping insignificant variables but in this case the authors do not test that this is the case. In any case, even if a variable is not statistically significant there is still no good reason to drop the variable in these types of analyses.
4. At most analysts should consider including the variable as a main effect as part of sensitivity analysis (even if they do not believe that the variable should be included as a main effect).

In their paper, Kotchen and Grant focus on the coefficient of the interaction, d, in this case which they use to support their claim that DST increases energy usage. My guess is that if they were to estimate the model correctly, the size of the coefficient, d, would fall. Right now their estimates of d are partially capturing the effects of the omitted variable. I suppose the other possibility is that including all the relevant variables as main effects could have resulted in some perfect collinearity although they don't indicate this is the case.