1. Test the null that two alternatives are the same (i.e. the mean difference is zero)
2. If the null is not rejected this does not imply that we accept the null that the two alternatives are the same. All we can say is that the two alternatives are not different, which is not the same thing as saying that it is the same. This is the conservative interpretation that was drilled into us in graduate school. (Splitting hairs or angels dancing on a pin?)
3. If the null is rejected, then we can say that the two alternatives are different. In fact, we can say that the two alternatives are not the same. But, can we conclude that one is better than the other, i.e. if the mean difference is positive?
This is what I am wrestling with when reading:
Early Education Policy Alternatives: Comparing Quality and Outcomes of Head Start and State Prekindergarten by Gary T Henry, Craig S Gordon and Dana K Rickman
In the paper, I conclude that the quality difference between state pre kindergarten programs are Head Start programs are different, i.e. we can reject the null that they are the same. (See paper for various measures of qualities and outcomes. For instance, kids in pre kindergarten do better in standardized tests a couple of years later than Head Start kids, Head Start centers/programs do not have as many teachers with BA as state pre-K programs, etc.)
In fact the differences are positive on the side of state pre-K programs but instead of concluding that these programs are better than Head Start, the authors choose to word it as follows (from the abstract, emphasis mine):
The two groups were statistically similar at the beginning of their preschool year on three of four direct assessments (p less than 0.05), but by the beginning of kindergarten the children attending the state prekindergarten program posted higher developmental outcomes on five of six direct assessments (p less than 0.05) and 14 of 17 ratings by kindergarten teachers (p less than 0.05). This study indicates that economically disadvantaged children who attended Georgia's universal prekindergarten entered kindergarten at least as well prepared as similar children who attended the Head Start program.
Can we not conclude that kids entering state pre-K programs are better off than Head Start kids? Or, that Head Start kids are worse off than prekindergarten kids?