“As a professional psychology researcher, it’s my job to lie to the participants in my experiments so I can lie to others with statistics using their data”. -On understanding the role of deception in psychology research
In my last post, I discussed the topic of fear: specifically, how social and political agendas can distort the way people reason about statistics. The probable function of such distortions is to convince other people to accept a conclusion which is not exactly well supported by the available evidence. While such behavior is not exactly lying – inasmuch as the people making these claims don’t necessarily know they’re engaged in such cognitive distortions – it is certainly on the spectrum of dishonesty, as they would (and do) reject such reasoning otherwise. In the academic world, related kinds of statistical manipulations go by a few names, the one I like the most being “researcher degrees of freedom“. The spirit of this idea refers to the problem of researchers selectively interpreting their data in a variety of ways until they find a result they want to publish, and then omit mentioning all the ways that their data did not work out, or might be interpreted. On that note, here’s a scary statistic: 1-in-3 college men would rape a woman if they could get away with it. Fortunately (or unfortunately, depending on your perspective) the statistic is not at all what it seems.
“…But the researchers failed to adequately report their methods! Spooky!”
The paper in question (Edwards et al, 2014) seeks to try and understand the apparent mystery behind the following finding: when asked if they ever raped anyone, most men will say “no”; when asked instead whether they ever held someone down to coerce them into having sex, a greater percentage of men will indicate that they have. Women’s perceptions about the matter seem to follow suit. As I wrote when discussing the figure that 25% of college women will be raped:
The difference was so stark that roughly 75% of the participants that Koss had labeled as having experiencing rape did not, themselves, consider the experience to be rape.
What strikes me as curious about these findings is not the discrepancy in responses; that much can likely be explained by positing that these questions are perceived by the participants to be asking about categorically different behaviors. After all, if they were actually perceived to be asking about the same thing, you would see a greater agreement between the responses of both men and women between questions, which we do not. Instead, the curious part is that authors – like Edwards et al (2014) – continue to insist that all those participants must be wrong, writing, “…some men who rape do not seem to classify their behavior as such” (Jesse Singal at NYmag.com expresses a similar view, writing: “At the end of the day, after all, the two groups are saying the exact same thing“). Rather than conclude there is something wrong with the questions being asked (such as, say, they are capturing a portion of the population who would have rough, but consensual sex), they instead conclude there is something wrong with everyone else (both men and women) answering them. This latter explanation strikes me as unlikely.
There’s already something of a bait-and-switch taking place, then, but this is far from the only methodological issue involved in deriving that scary-sounding 1-in-3 figure. Specifically, Edwards et al (2014) asked their 86 male participants to fill out part of the “attraction to sexual aggression” scale (Malamuth, 1989). On this scale, participants are asked to indicate, from 1 to 5, how likely they would be to engage in a variety of behaviors, with a “1″ corresponding to “not likely at all”, while “5″ corresponds to “very likely”. Included on this scale are two questions, one concerning whether the respondent would “rape” a woman, and another asking about whether he would “force her to do something she did not want to do” in a sexual setting. The participants in question were asked about their likelihood of engaging in such behaviors “if nobody would ever know and there wouldn’t be any consequences”. Edwards et al (2014) report that, if such criteria were met, 31% of the men would force a woman to do something sexually, whereas only 13% would rape a woman.
If you’re perceptive, you might have noticed something strange already: that 1-in-3 figure cannot be straightforwardly derived from the sexual aggression scale, as the scale is a 5-point measure, whereas the 1-in-3 statistic is clearly dichotomous. This raises the question of how one translates the scale into a yes/no response format. Edwards et al (2014) do not explicitly mention how they managed such a feat, but I think the answer is clear from the labeling in one of their tables: “Any intention to rape a woman” (emphasis, mine). What the researchers did, then, was code any response other than a “1″ as an affirmative; the statistical equivalent of saying that 2 is closer to 5 than it is to 1. In other words, the question was, “Would you rape a woman if you could get away with it”, and the answers were, effectively, “No, Yes, Yes, Yes, or Yes”. Making the matter even worse is that all that participants were answering both questions. This means they saw a question asking about “rape” and another question about “forcing a woman to do something she didn’t want to”. As participants likely figured that there was no reason the researchers would be asking the same question twice, they would have very good reason for thinking that these questions refer to categorically different things. For the authors to then conflate the two questions after the fact as being identical is stunningly disingenuous.
“The problem isn’t me; it’s everyone else”
To put these figures in better context, we could consider the results reported by Malamuth (1989). In response to the “Would you rape if you wouldn’t get caught” question, 74% of men indicated “1″ and 14% indicated a “2″, meaning a full 88% of them fell below the midpoint of the scale; by contrast, only 7% fell above the midpoint, with about 5% indicating a “4″ and 2% indicating a “5″. Of course, reporting that “1-in-3 men would rape” if they could get away with it is much different than saying “less than 1-in-10 probably would”. The authors appear interested in deriving the most-damning interpretation of their data possible, however, as evidenced by their unreported and, in my mind, unjustifiable grouping of the responses. That fact alone should raise alarm bells as to whether the statistics they provide you would do a good job of predicting reality.
But let’s go ahead and take these responses at face value anyway, even if we shouldn’t: somewhere between 10-30% of men would rape a woman if there were no consequences for doing so. How alarming should that figure be? On the first front, the hypothetical world of “no consequence” doesn’t exist. Some proportion of men who would be interested in doing such things are indeed restrained from doing so by the probability of being punished. Even within that hypothetical world of freedom from consequences, however, there are likely other problems to worry about, in that you will always find some percentage of the population willing to engage in anti-social behavior that harms others when there are no costs for doing so (in fact, the truly strange part is that lots of people indicate they would avoid such behaviors).
Starting off small, for instance, about 70% of men and women indicate that they would cheat on their committed partner if they wouldn’t get caught (and slightly over 50% have cheated in spite of those possible consequences). What about other acts, like stealing, or murder. How many people might kill someone else if there would be no consequences for it? One informal poll I found placed that number around 40%; another puts it a little above 50% and, when broken up by sex, 32% of women would and a full 68% of men would. Just let those numbers sink in for a moment: comparing the two numbers for rape and murder, the men in Edwards et al (2014) were in between 2-to-7 times less likely to say they would rape a woman than kill someone if they could, depending on how one interprets their answers. That’s a tremendous difference; one that might even suggest that rape is viewed as a less desirable activity than murder. Now that likely has quite a bit to do with some portion of that murder being viewed as defensive in nature, rather than exploitative, but it’s still some food for thought.
There are proportionately fewer defensive rapes than defensive stabbings…
This returns us nicely to the politics of fear. The last post addressed people purposefully downplaying the risks posed by terrorist attacks; in this case, we see people purposefully inflating the reported propensities to rape. The 1-in-3 statistic is clearly crafted in the hopes of making an issue seem particularly threatening and large, as larger issues tend to have more altruism directed towards them in the hopes of a solution. As there are social stakes in trying to make one’s problems seem especially threatening, however, this should immediately make people skeptical when dealing with such statistics for the same reasons you shouldn’t let me tell you about how smart or nice I am. There is a very real risk of artificially trying to puff one’s statistics up, as people might come to eventually start not trusting you about things as the default, even for different topics entirely; this should hold true especially if they belong to a group targeted by such misleading results. The undesirable outcomes of such a process being, rather than increases in altruism and sympathy devoted to a real problem, apathy and hostility. Lessons learned from fables like The Boy Who Cried Wolf are timely as ever, it would seem.
References: Edwards, S., Bradshaw, K., & Hinsz, V. (2014). Denying rape but endorsing forceful intercourse: Exploring differences among responders. Violence & Gender, 1, 188-193.
Malamuth, N. (1989). The attraction to sexual aggression scale: Part 1. The Journal of Sex Research, 26, 26-49.