Data scientist(?)

One of the apparently ‘hot’ jobs these days is the data scientist. So hot in fact that the Harvard Business Review has named it the sexiest job of the 21st century. I came across Rachel Schutt’s Data Science class at Columbia (HT: Andrew Gelman) and she has a description of what a data scientists does (or should do):

What is a Data Scientist?

Let me start with academia because that’s quicker. Then industry.

In Academia: No one calls themselves a Data Scientist yet in universities. There are 60 students in my class from across disciplines. I thought when I proposed the course it would be statisticians, applied mathematicians and computer scientists who showed up. Actually it’s them plus sociologists, journalists, political scientists, biomedical informatics students, students from NYC government agencies and non-profits related to social welfare, someone from the architecture school, environmental engineering, pure mathematicians, business marketing students, and students who already work as data scientists. Am I missing someone? They’re all interested in figuring out ways to solve important problems, often of social value, with data.

For the term Data Science to catch on in academia at the level of the faculty, the research area needs to be more formally defined. I see a rich set of problems that could be many PhD theses. My current working definition is a Data Scientist in this setting is a Scientist (from social scientists to biologists) who work with large amounts of data, and must grapple with computational problems posed by the structure, size, messiness and nature of the data, while simultaneously solving a real world problem. Across academic disciplines, the computational and deep data problems are the same. So if researchers across departments join forces, they can solve multiple real-world problems from different domains.

In Industry:

It depends on the level of seniority and whether you’re talking about the internet industry in particular. The role of data scientist need not be exclusive to the tech world, but that’s where the term originated so for the purposes of the conversation, let me say what it means there:

A Chief Data Scientist should be setting the data strategy of the company which involves a variety of things: setting everything up from the engineering and infrastructure for collecting data and logging, to privacy concerns; deciding what data will be user-facing, how data is going to be used to make decisions, and how it’s going to be built back into the product. She should manage a team of engineers, scientists and analysts and she should communicate with leadership across the company including the CEO, CTO and product leadership. She’ll also be concerned with patenting innovative solutions, and setting research goals.

More generally, a data scientist is someone who knows how to extract meaning from and interpret data, which requires both tools and methods from statistics and machine learning, as well as being human. She spends a lot of time in the process of collecting, cleaning and munging data, because data is never clean. This process requires persistence, statistics and software engineering skills– skills that are also  necessary for understanding biases in the data, and for debugging logging. Once she gets the data into shape, a crucial part is exploratory data analysis which combines visualization and data sense. She’ll find patterns, build models and algorithms, some with the intention of understanding product usage and the overall health of the product, and others serve as prototypes that ultimately get baked back into the product. She may design experiments, and is a critical part of data-driven decision making. She’ll communicate with team members, engineers, and leadership in clear language and using data visualizations so that even if her colleagues are not immersed in the data themselves, they will understand the implications.

Looking at the syllabus it sure sounds a lot like data mining. I guess being a scientist beats being a miner.

Where the jobs are

Census has been touting its web site On the Map for some time now and so I decided to take a look based on some blog posts by Sirinya Tritipeskul, particularly this one that looks at jobs along a proposed subway route in Century City, Los Angeles. What was impressive was the ability to map job density all the way along the proposed route.

I was more interested in other aspects of the data - for instance, can it answer the following questions:
1. What has happened to employment in the brewery industry in Milwaukee in the past 3 years (the data is only from 2007-2009)?
2. What would happen to commuting patterns if Montgomery County's largest employer moved to Prince George's County?

Sadly, the data does not allow for such fine grain analysis. Industries only go down to major groups. What would be awesome would be to be able to click on the major industries and drill down to maybe the top 5 of the smaller industries the are part of the major group. Then clicking on the finer category of industries would give the names of the top 3 employers in that geographic region.

The use of maps to present the data is interesting only because it would not have been the first thing I would have thought of as a way to deliver the data.

Average monthly condo fees in the U.S.

Data source:

Steven Ruggles, Matthew Sobek, Trent Alexander, Catherine A. Fitch, Ronald Goeken, Patricia Kelly Hall, Miriam King, and Chad Ronnander. Integrated Public Use Microdata Series: Version 4.0 [Machine-readable database]. Minneapolis, MN: Minnesota Population Center [producer and distributor], 2008.

We've been looking at condos around the DC area and it got me wondering as to what the average condo fees were like around the country. The above is a chart of the average condo fees tabulated from IPUMS. Because of topcoding the average is below the true average. The mean and max are labeled at the end of the bars. Unfortunately, I'm unable to improve on the resolution of the jpeg.

The SAS code for this is:

proc means data = cf.hh2008 N mean std min max;
where year = 2008 and condofee > 0;
class stateicp;
var condofee;
weight hhwt;
output out = summ mean = meancondofee min = mincondofee max=maxcondofee;
run;

data summ2;
set summ;
proc sort; by descending meancondofee;
run;

data annosum;
set summ2;
%annomac;
%dclanno;
%system(2,2,3);
midpoint = stateicp;
x = round(meancondofee,1);
lbl = compress(put(meancondofee, 8.)"/"put(maxcondofee, 8.));
%label(x,.,lbl,black,0,0,1.8,swissb,6);
run;

filename grafout 'chart1.jpeg';
goptions device = jpeg targetdevice = jpeg ftext="Verdana" gsfname = grafout xmax=8 in ymax=6 in xpixels=4000 ypixels=3000 vpos=40 hpos=40 lfactor=1 ;
axis1 label = none;
title1 h=1 'Average monthly condominium fees in 2008: U.S. States';
proc gchart data=summ2;
where _type_ ^=0;
hbar stateicp / sumvar=meancondofee maxis=axis1 discrete descending nostat anno=annosum(where=(_type_^=0));
run;
quit;

What is the current crisis called?

Subprime crisis, credit crisis, or the plain generic financial crisis? According to Google Trends financial crisis (yellow) is ranked highest while subprime crisis (blue) and credit crisis (red) are almost tied. The green is "Great Depression" which has a surprising pattern.

Overlaying/superimposing NBER recession dates on time series

I don't really know what you'd call it but I've always been curious how Econbrowser produces nice charts like this one on recession probabilities. I'm a SAS geek and this is how I coded it although I think that SAS should have an easier way of doing this without having to use an ANNOTATE data set which despite having used it quite a lot on project work is still a little bit of a mystery to me. Unfortunately, I don't know enough about HTML to render the code correctly so the input statement looks wrong.

data NBERdates;
   length PeakQ $ 20 TroughQ $ 20;
   informat Peak Trough ddmmyy10.;
   format Peak Trough date9.;
   input PeakQ 1-20 TroughQ 21-40 Peak Trough;
datalines;
February 1945(I)    October 1945 (IV)   1/2/1945        1/10/1945
November 1948(IV)   October 1949 (IV)   1/11/1948       1/10/1949
July 1953(II)       May 1954 (II)       1/7/1953        1/5/1954
August 1957(III)    April 1958 (II)     1/8/1957        1/4/1958
April 1960(II)      February 1961 (I)   1/4/1960        1/2/1961
December 1969(IV)   November 1970 (IV)  1/12/1969       1/11/1970
November 1973(IV)   March 1975 (I)      1/11/1973       1/3/1975
January 1980(I)     July 1980 (III)     1/1/1980        1/7/1980
July 1981(III)      November 1982 (IV)  1/7/1981        1/11/1982
July 1990(III)      March 1991(I)       1/7/1990        1/3/1991
March 2001(I)       November 2001 (IV)  1/3/2001        1/11/2001
;
run;

data NBERdates;
    set NBERdates;
    where Peak > '01Jan1947'd and Peak < '31Dec2000'd;
    PeakS = "'"put(Peak, date9.)"'d";
    TroughS = "'"put(Trough, date9.)"'d";
run;

filename gdp 'GDPC1.txt'; /* File downloaded from FRED St. Louis */
data GDP;
   infile gdp firstobs = 14;
   /* Read two variables: date and GDP */
   input date value;
   informat date yymmdd10.;
   format date yymmdd10.;
   lngdp = log(value);
   /* We will refer to Y instead of LNGDP for consistent notation */
   y=lngdp;
   /* Calculate the first difference of y */
   diffy = dif(y);
run;

%annomac;

data NBERDate1;
    set NBERdates;
    date = Peak; output;
    date = Trough; output;
run;

data anno;
    merge GDP(rename = (y=y1)) NBERdate1; by date;
    RETAIN YSYS XSYS '2';
    %bar(Peak,7.3,Trough,9.2,ligr,0,S);
run;

goptions ftext = swiss fontres = presentation htext = 0.9;
axis1 label = (angle=90);
axis2 order = ('1Jan1945'd to '30Jun2000'd by year5) minor = none;
symbol1 color = red interpol = j;
symbol2 color = black interpol = j;

proc gplot data = GDP;
plot lngdp * date /legend vaxis = axis1 haxis = axis2 annotate = anno chref=blue;
format date yyq6.;
run;
quit;

Excel charts for data visualization and exploration

Along the way from the previous post, I also came across an interesting discussion of charting EU household expenditures. Following the thread, I discovered Excel was able to do this and this. Some tips here and here.

Graphing Stock Market Crashes for Comparison

The Big Picture posted a chart that compared 4 stock market crashes.
My first thoughts were:
1. The vertical axes are not the same so it looks deceptive.
2. The horizontal axes are not the same so again, it may look deceptive.





This tries to replicate the flavor of the original charts (but perhaps not very sucessfully).




But thinking about it a little more I thought perhaps it was all right. After all, a 500 point drop today is a lot different than a 500 point drop 30 years ago. But I'm still not convinced.

What if the index was plotted using a logscale? Would that make the market crashes more comparable?