Data versus anecdotes

Andrew Gelman had an interesting post on data Skepticism About Empirical Studies:

Nick Firoozye writes,

I [Firoozye] wanted to point your attention to the following podcast by Ian Ayres on Supercrunchers, where he shows himself an enthusiastic (if perhaps a bit naïve) proponent of the statistical method. Entertaining, definitely. One thing though that I thought you might be interested in is Russ Roberts’ (the interviewer's) own skepticism over the econometric method, which I think probably warrants a response. It may be that Roberts’ own view is due to his now-Austrian economics slant (i.e., somewhat anti-formallist approach) or perhaps to the fact that mainstream econometrics is a frequentist pursuit and one might question the honesty of the results as a consequence.

I don't really have much to add here, except that the problem noted by Roberts (it's hard to know whether to believe a statistical study) is even more of a problem with non-statstical empirical studies (i.e., anecdotes). ... But there are a lot of areas where we have only weak opinions which can indeed be swayed by data (see here for some examples). These cases are important in their own right and also can serve as benchmarks for the success of statistical analysis, so that we can trust good analyses more when they're applied to tougher problems. This is one way that applied statistics proceeds, by exemplary analyses of problems that might not be hugely important on their own terms but serve as useful templates.

Personally, I don't find one empirical study all too convincing (whether it is an econometric study or a randomized trial). But over time as more and more studies on different data sets start to show the same result then I would be swayed. I also would place more emphasis on anecdotes than Prof. Gelman does. These stories can be extremely convincing even though they may not be representative of the situation being discussed. As pointed out by McCloskey, convincing others comes down to who can tell a better story. A study with a good anecdote beats everything, but I think good anecdotes can beat a statistical study any day. (Notice how hard economists have been trying to convince the public that free trade is good but the personal stories from those who have lost jobs over trade always trumps these arguments.)

Another well cited example is Ronald Reagan's characterization of those on welfare as Welfare queens. This label while not representative I think had a negative effect on everyone on welfare.