In the course of proposing new research and getting input from others, I have had multiple researchers raise the same basic concern to me: the project I’m proposing might be unlikely to eventually get published because, given that I find the results I predict that I will, reviewers might feel the results are not interesting or attention-grabbing enough. While I don’t doubt that the concern is, to some degree, legitimate*, it has me wondering about whether their exists an effect that is essentially the reverse of that issue. That is, how often does bad research get published simply on the grounds that it appears to be interesting, and are reviewers willing to overlook some or all the flaws of a research project because it is, in a word, cute?
Which is why I always make sure my kitten is an author on all my papers.
The cute experiment of the day is Simons & Levin (1998). If you would like to see a firsthand example of the phenomenon this experiment is looking at before I start discussing it, I’d recommend this video of the color changing card trick. For those of you who just want to skip right to the ending, or have already seen the video, the Simons & Levin (1998) paper sought to examine “change blindness”: the frequent inability of people to detect changes in their visual field from one moment to the next. While the color changing card trick only replaced the colors of people’s shirts, tablecloths, or backdrops, the experiment conducted by Simons & Levin (1998) replaced actual people in the middle of a conversation to see if anyone would notice.The premise of this study would appear to be interesting on the grounds that many people might assume that they would notice something like the fact that they were suddenly talking to a different person then they were a moment prior, and the results of this study would seem to suggest otherwise. Sure sounds interesting when you phrase it like that.
So how did the researchers manage to pull off this stunt? The experiment began when a confederate holding a map approached a subject on campus. After approximately 10 or 15 seconds of talking, two men holding a door would pass in between the confederate and the subject. Behind this door was a second confederate who changed places with the first. The second confederate would, in turn, carry on the conversation as if nothing had happened. Of the 15 subjects approached in such a manner, only 7 reported noticing the change of confederate in the following interview. The authors mention that out of the 7 subjects that did notice the change, there seemed to be a bias in age: specifically, the subjects in the 20-30 age range (which was similar to that of the confederates) seemed to notice the change, whereas the older subjects (in the 35-65 range) did not. To explain this effect, Simons & Levin (1998) suggested that younger subjects might have been treating the confederates as their “in-group” because of their age (and accordingly paying more attention to their individual features) whereas the older subjects were treating the confederates as their “out-group”, also because of their age (and accordingly paying less attention to their features).
In order to ostensibly test their explanation, the authors ran a follow-up study. This time the same two confederates were dressed as construction workers (i.e. they wore slightly different construction hats, different outfits, and different tool belts) in order to make them appear as more of an “out-group” member to the younger subjects. The confederates then exclusively approached people in the younger age group. Lo and behold, when the door trick was pulled, this time only 4 of the 12 subjects caught on. So here we have a cute study with a counter-intuitive set of results and possible implications for all sorts of terms that end in -ism.
And the psychology community goes wild!
It seems to have gone unnoticed, however, that the interpretation of the study wasn’t particularly good. The first issue, though perhaps the smallest, is the sample size. Since these studies only ran a total of 13.5 subjects each, on average, the extent to which this difference in change blindness (approximately 15%) across groups is just due to chance is unknown. Let’s say, however, that we give the results the benefit of the doubt and assume that they would remain stable if sample size was scaled up. Even given that consideration, there are still some very serious problems remaining.
The larger problem is that the authors did not actually test their explanation. This issue comes in two parts. First, Simons and Levin (1998) proposed that subjects were using cues of group membership in determining whether or not to pay attention to an individual’s features. In their first study, this cue was assumed to be age; in the second study, this cue was assumed to now be construction worker. Of note, however, is that the same two confederates took part in both experiments, and I doubt their age changed much between the two trials. This means that if Simons and Levin (1998) were right, age only served as an indicator of group membership in first context; in the second, that cue was overridden by another – construction worker. Why that might be the case is left completely untouched by the authors, and that seems like a major oversight. The second part is that the authors didn’t test whether the assumed “in-group” would be less change blind. In order to do that they would have had to, presumably, pull the same door trick using construction workers as their subjects. Since Simons and Levin (1998) only tested an assumed out-group, they are unable to make a solid case for differences in group membership being responsible for the effect they’re talking about.
Finally, the authors seem to just assume that the subjects were paying attention in the first place. Without that assumption these results are not as counter-intuitive as they might initially seem, just as people might not be terribly impressed by a magician who insisted everyone just turned around while he did his tricks. The subjects had only known the confederates for a matter of seconds before the change took place, and during those seconds they were also focused on another task: giving directions. Further, the confederate (who is still a complete stranger at this point) is swapped out for another very similar one (both are male, both are approximately the same age, race, and height, as well as being dressed very similarly). If the same door trick was pulled with a male and female confederate, or a friend and a stranger, or people of different races, or people of different ages, and so on, one would predict you’d see much less change blindness.
My only change blindness involves being so rich I can’t see bills smaller than $20s
The real interesting questions would then seem to be what cues to people attend to, why do they attend to them, and in what order are they attended to? None of these questions are really dealt with by the paper. If the results they present are to be taken at face value, we can say the important variables are often not the color of one’s shirt, the sound of one’s voice (within reason), very slight differences in height, and modestly different hairstyles (when one isn’t wearing a hat) when dealing with complete strangers of similar gender and age, while also involved in another task.
So maybe that’s not a terribly surprising result, when phrased in such a manner. Perhaps the surprising part might even be that so many people noticed the apparently not so obvious change. Returning to the initial point, however, I don’t think many researchers would say that an experiment designed to demonstrate that people aren’t always paying attention to and remembering every single facet of their environment would be a publishable paper. Make it cute enough, however, and it can become a classic.
*Note: whether the concerns are legitimate or not, I’m going to do the project anyway.
References: Simons, D.J., & Levin, D.T. (1998). Failure to detect changes to people during a real-world interaction Psychonomic Bulletin & Review, 5, 644-649 DOI: 10.3758/BF03208840
Wow I didn’t notice the change at all in “the color changing card trick”. It really makes me wonder what else in my day I am not noticing. I was actually trying hard to watch for a color change, but maybe I was just expecting the cards to be changing so my mind wasn’t looking for anything else. All I know is that this is an interesting matter I’d personally like to know more about.
Perhaps a little historical context would help to explain why this study was published (beyond just being cute). Keep in mind that it appeared nearly 15 years ago (in 1998), in what were the earliest days of research on change blindness. Although people had studied change detection with simple displays for a while before then, only in the late 1990s were these findings extended to photographs and videos (the first papers on the topic using photographs or videos weren’t published until 1996). At the time, there was a lot of skepticism about whether or not such findings would generalize to real-world perception and memory. In fact, we conducted that study after talking with Ulric Neisser (one of the founders of the field of cognitive psychology, and probably the most important figure in ecological approaches to cognition). He argued that our earlier results with videos would NOT generalize to a real-world setting. The study was conducted to test whether those results would generalize. For what it’s worth, although we predicted that people should miss the change (based on our views of visual awareness), we didn’t think the real-world change would work either.
In that context and time, the finding that people could miss a change to a person they were talking to was novel and important because it showed the generality of the phenomenon of change blindness. To this day, most people, when asked to predict what would happen, are convinced that they would notice the change—it is inconsistent with intuitions. Of course, for researchers or academics with the benefit of 15 years of hindsight into research on failures of awareness, it might not seem surprising any more.
The study was admittedly a “demo” study—it was effectively an existence proof of the idea that change blindness would occur during a real interaction. The sample sizes are small by modern standards, and would not be deemed large enough now. That said, the result is easy to replicate and variants of it have been conducted many times, both by me and my collaborators and by undergraduates and graduate students. Follow-up papers have tended to use larger samples, many more conditions, and more systematic manipulations of possible moderator variables. This was the first study of its type.
In the paper, the in-group/out-group hypothesis was one of several possible explanations for the pattern fo results. A full design would have used older experimenters and younger experimenters approaching older and younger subjects, but that proved impractical (it was hard to find older experimenters who had the time to run the task but weren’t already well known to pedestrians in that part of campus). The finding that the same change (actually, in some ways, a bigger one given the clothing differences) was less noticed by the same cohort of subjects shows the power of categorization. In the study, young subjects noticed 100% of the time when the changed people looked like students, but only about 1/3 of the time when the the *same two people* were dressed as construction workers. Might other explanations besides in-group/out-group coding hold? Yes. But again, this was the first demonstration of it’s type, and there aren’t too many manipulations in cognitive psychology that induce a 60+% change in noticing rates.
Would this study be published now? Almost certainly not. It’s cute, but in the context of nearly 15 years of additional work on change blindness, it would no longer be of sufficient interest as a “demo” study. It’s a bit misleading to use this study as an example of one that was published *because* it was cute. Cuteness didn’t hurt, but that wasn’t the reason it was published. It was also empirically and theoretically important *at the time it was published* and in the context of what had come before it in the change blindness literature.
Sometimes a demo study of this sort can be important and influential. For that to occur, the result must be clear and robust (as it is in this case) and there must be strong a-priori reasons (or at least beliefs) that it might not work. Making a paper more engaging for readers (via cuteness, humor, etc) isn’t inherently a bad thing. But, the reason it should be published is not the cuteness but the point it makes. I think the reason this paper has been so heavily cited is because it shows just how well the earlier results generalized to a more real situation and because the result continues to be surprising to those unfamiliar with the limits of attention. It is not a great test of your hypothesis that cuteness alone contributes to notoriety.
It’s a pleasure to have you commenting here, Dan. On your first point, yes, the study most certain does demonstrate that the phenomenon of change blindness does generalize to the world outside of the computer screen. I would be curious as to why Ulric thought it wouldn’t, given that we (or at least some parts of our brain) treat images on screens and paper as, more or less, real. If we didn’t, our fascination with watching TV or looking at pictures would certainly be a curious habit. Had the only conclusion been that the effect generalizes, I would have no issue with the interpretation.
As for the matter of people’s intuitions concerning whether they would notice a change, those intuitions are certainly worthy of a thorough consideration. I get the sense that, yes, most people would probably report that they feel generally aware of their surroundings and they’re not oblivious to the point that they feel they would mistake one person for another. Saying that you were the kind of person who would miss the substitution of one person for another generally would likely not reflect very positively on one’s character. Precisely how the subjects interpreted such a question, however, might be another matter: did they imagine a very similar looking person in just a slightly different set of clothes (two easy-to-mistake people) or did they imagine two people who looked radically different? Did they assume they were already paying attention to the person they were talking to, as many of the participants in the change blindness research likely weren’t? Depending on how one phrases the question, I would imagine you might see different set of intuitions.
I am aware that further studies had been conducted since, but my main focus is on the information in the paper as presented. On the basis of that presented data, the interpretations that were put forth didn’t seem to be well substantiated. I’m not saying they’re necessarily wrong, just that the conclusions aren’t able to made given that data set. My comments about the cuteness of the study relate to that point, specifically: were some of the shortcomings of the research overlooked because the results were not intuitive to people and/or because the experimental method was interesting? Admittedly, I don’t have an answer to this question.
Don’t get me wrong; I think there is a value in the method that this study used. I feel it would be well adapted for studying questions like the ones I posed in the post (in fact, I assume some people have already done such research, though I am unfamiliar with it). For instance, people could probably tell you that a white male construction worker approximately in their 20s or 30s had asked them for directions without being able to recall the color of their shirt, the sound of their voice, or accurately pick their face out of a lineup. Figuring out which details people attend to, along with what order they tend to be attended to in and why, could lead to many productive avenues of research, and this kind of design would be a good method of assessment.
As an aside, the construction worker manipulation reminded me of this paper: http://www.psych.ucsb.edu/research/cep/papers/eraserace.pdf. Giving someone a badge of group membership – in this case a construction hat – may well trigger mechanisms designed for detecting social coalitions. In fact, giving someone a badge like that may increase the tendency of subjects to be blind to other changes in the visual field, as badges may tend to command attention at the expense of other factors. That is, of course, provided the badges remain constant; were you to swap one badge for another – i.e. replacing the construction worker confederate with one wearing a yamaka – you would predict that people would then be less change blind.
Pingback: Dinner, With A Side Of Moral Stances | Pop Psychology