The Trouble With Studies

I'm reading the book Blink: The Power of Thinking Without Thinking by Malcolm Gladwell. It's a popular science book whose subject is exactly what the subtitle says that it is. It's not bad, though I wouldn't particularly recommend it. There's not enough depth of information to suit my tastes.

However, Gladwell talks about one particular study that really struck me. A couple of psychologists gave black college students the Graduate Record Exam, the test that is used for entry into graduate school. When the students were asked to identify their race on a pretest questionnaire, the number of items that they got correct was cut in half. For those of you who might like to check out the study yourselves, here's the reference. I haven't done so myself. Claudia Steele & Joshua Aronson, "Stereotype Threat and Intellectual Test Performance of African Americans," Journal of Personality and Social Psychology 69, no. 5, (1995), pp 797-811.

It seems remarkable to me that the effect could be so dramatic. No doubt negative racial and ethnic stereotypes have deleterious effects on the subject groups, but this is hard to believe. I should probably check the study myself, just to see what the protocols were. After thinking about it for a few minutes, it occurs to me that there are a number of factors that could affect the outcome of the experiment. For example, what was the race of those who administered the questionnaire and the test? That could have an effect. However, there is something even more subtle that could have influenced the result. The black students undoubtedly knew that they were taking part in an experiment. Ethics and legal liability suggest that this must have been the case. Hence, it is conceivable that something like the following could have been going on. The students who were not given the questionnaire would undoubtedly think that their performance on the test was the point of the experiment. Those students who were given the questionnaire might have come to the conclusion that the experiment was about something altogether different and they might not have taken the GRE as seriously. I'm not saying that this is the way that it was. I'm just saying that these are possibilities, among others, that have to be eliminated.

I don't mean to belittle the effect of racial and ethnic stereotypes. They undoubtedly have deleterious effects. It's just that these studies often aren't what they seem to be. In order to decide whether the effect reported by this study is real or an artifact would require several experiments. This one study doesn't mean much.