Getting To Know Your Outliers: More About Video Games

As I mentioned in my last post, I’m a big fan of games. For the last couple of years, the game which has held the majority of my attention has been a digital card game. In this game, people have the ability to design decks with different strategies, and the success of your strategy will depend on the strategy of your own opponent; you can think of it as a more complicated rock-paper-scissors component. The players in this game are often interested in understanding how well certain strategies match up against others, so, for the sake of figuring that out, some have taken it upon themselves to collect data from the players to answer those questions. You don’t need to know much about the game to understand the example I’m about to discuss, but let’s just consider two decks: deck A and deck B. Those collecting the data managed to aggregate the outcome of approximately 2,200 matches between the two and found that, overall, deck A was favored to win the match 55% of the time. This should be some pretty convincing data when it comes to getting a sense for how things generally worked out, given the large sample size.

Only about 466 more games to Legend with that win rate

However, this data will only be as useful to us as our ability to correctly interpret it. A 55% success rate captures the average performance, but there is at least one well-known outlier player within the game in that match. This individual manages to consistently perform at a substantially higher level than average, achieving wins in that same match up around 70-90% of the time across large sample sizes. What are we to make of that particular data point? How should it affect our interpretation of the match? One possible interpretation is that his massively positive success rate is simply due to variance and, given enough games, the win rate of that individual should be expected to drop. It hasn’t yet, as far as I know. Another possible explanation is that this player is particularly good, relative to his opponents, and that factor of general skill explains the difference. In much the same way, an absolutely weak 15-year-old might look pretty strong if you put him in a boxing match against a young child. However, the way the game is set up you can be assured that he will be matched against people of (relatively) equal skill, and that difference shouldn’t account for such a large disparity.

A third interpretation – one which I find more appealing, given my deep experience with the game – is that skill matters, but in a different way. Specifically, deck A is more difficult to play correctly than deck B; it’s just easier to make meaningful mistakes and you usually have a greater number of options available to you. As such, if you give two players of average skill decks A and B, you might observe the 55% win rate initially cited. On the other hand, if you give an expert player both decks (one who understands that match as well as possible), you might see something closer to the 80% figure. Expertise matters for one deck a lot more than the other. Depending on how you want to interpret the data, then, you’ll end up with two conclusions that are quite different: either the match is almost even, or the match is heavily lopsided. I bring this example up because it can tell us something very important about outliers: data points that are, in some way, quite unusual. Sometimes these data points can be flukes and worth disregarding if we want to learn about how relationships in the world tend to work; other times, however, these outliers can provide us valuable and novel insights that re-contextualize the way we look at vast swaths of other data points. It all hinges on the matter of why that outlier is one. 

This point bears on some reactions I received to the last post I wrote about a fairly-new study which finds no relationship between violent content in video games and subsequent measures of aggression once you account for the difficulty of a game (or, perhaps more precisely, the ability of a game to impede people’s feelings of competence). Glossing the results into a single sentence, the general finding is that the frustration induced by a game, but not violent content per se, is a predictor of short-term changes in aggression (the gaming community tends to agree with such a conclusion, for whatever that’s worth). In conducting this research, the authors hoped to address what they perceived to be a shortcoming in the literature: many previous studies had participants play either violent or non-violent games, but they usually achieved this method by having them play entirely different games. This means that while violent content did vary between conditions, so too could have a number of other factors, and the presence of those other factors poses some confounds in interpreting the data. Since more than violence varied, any subsequent changes in aggression are not necessarily attributable to violent content per se.

Other causes include being out $60 for a new controller

The study I wrote about, which found no effect of violence, stands in contrast to a somewhat older meta-analysis of the relationship between violent games and aggression. A meta-analysis – for those not in the know – is when a larger number of studies are examined jointly to better estimate the size of some effect. As any individual study only provides us with a snapshot of information and could be unreliable, it should be expected that a greater number of studies will provide us with a more accurate view of the world, just like running 50 participants through an experiment should give us a better sense than asking a single person or two. The results of some of those meta-analyses seem to settle on a pretty small relationship between violent video games and aggression/violence (approximately r = .15 to .20 for non-serious aggression, and about r = .04 for serious aggression depending on who you ask and what you look at; Anderson et a, 2010; Ferguson & Kilburn, 2010; Bushman et al, 2010), but there have been concerns raised about publication bias and the use of non-standardized measures of aggression.

Further, were there no publication bias to worry about, that does not mean the topic itself is being researched by people without biases, which can affect how data gets analyzed, research gets conducted, measures get created and interpreted, and so on. If r = .2 is about the best one can do with those degrees of freedom (in other words, assuming the people conducting such research are looking for the largest possible effect and develop their research accordingly), then it seems unlikely that this kind of effect is worth worrying too much about. As Ferguson & Kilburn (2010) note, youth violent crime rates have been steadily decreasing as the sale of violent games have been increasing (r = -.95; as well, the quality of that violence has improved over time; not just the quantity. Look at the violence in Doom over the years to get a better sense for that improvement). Now it’s true enough that the relationship between youth violent crime and violent video game sales is by no means a great examination of the relationship in question, but I do not doubt that if the relationship ran in the opposite direction (especially if were as large), many of the same people who disregard it as unimportant would never leave it alone.

Again, however, we run into that issue where our data is only as good as our ability to interpret it. We want to know why the meta-analysis turned up a positive (albeit small) relationship whereas the single paper did not turn up such a relationship, despite multiple chances to find it. Perhaps the paper I wrote about was simply a statistical fluke; for whatever reason, the samples recruited for those studies didn’t end up showing the effect of violent content, but the effect is still real in general (perhaps it’s just too small to be reliably detected). That seems to be the conclusion some responses I received contained. In fact, I had one commenter who cited the results of three different studies suggesting there was a casual link between violent content and aggression. However, when I dug up those studies and looked at the methods section, what I found was that, as I mentioned before, all of them had participants play entirely different games between violent and non-violent conditions. This messes with your ability to interpret the data only in light of violent content, because you are varying more than just violence (even if unintentionally). On the other hand, the paper I mentioned in my last post had participants playing the same game between conditions, just with content (like difficulty or violence levels) manipulated. As far as I can tell, then, the methods of the paper I discussed last week were superior, since they were able to control more, apparently-important factors.

This returns us to the card game example I raised initially: when people play a particular deck incorrectly, they find it is slightly favored to win; when someone plays it correctly they find it is massively favored. To turn that point to this analysis, when you conduct research that lacks the proper controls, you might find an effect; when you add those controls in, the effect vanishes. If one data point is an outlier because it reflects research done better than the others, you want to pay more attention to it. Now I’m not about to go digging through over 130 studies for the sake of a single post – I do have other things on my plate – but I wanted to make this point clear: if a meta-analysis contains 130 papers which all reflect the same basic confound, then looking at them together makes me no more convinced of their conclusion than looking at any of them alone (and given that the specific studies that were cited in response to my post all did contain that confound, I’ve seen no evidence inconsistent with that proposal yet). Repeating the same mistake a lot does not make it cease to be a mistake, and it doesn’t impress me concerning the weight of the evidence. The evidence acquired through weak methodologies is light indeed.  

Research: Making the same mistakes over and over again for similar results

So, in summation, you want to really get to know your data and understand why it looks the way it does before you draw much in the way of meaningful conclusions from it. A single outlier can potentially tell you more about what you want to know than lots of worse data points (in fact, it might not even be the case that poorly-interpreted data is recognized as such until contrary evidence rears its head). This isn’t always the case, but to write off any particular data point because it doesn’t conform to the rest of the average pattern – or to assume its value is equal to that of other points – isn’t always right either. Meeting your data, methods, and your measures is quite important for getting a sense for how to interpret it all. 

For instance, it has been proposed that – sure – the relationship between violent game content and aggression is small at best (there seems to be some heated debate over whether it’s closer to r = .1 or .2) but it could still be important because lots of small effects can add up over time into a big one. In other words, maybe you ought to be really wary of that guy who has been playing a violent game for an hour each night for the last three years. He could be about to snap at the slightest hint of a threat and harm you…at least to the extent that you’re afraid he might suggest you listen to loud noises or eat slightly more of something spicy; two methods used to assess “physical” aggression in this literature due to ethical limitations (despite the fact that, “Naturally, children (and adults) wishing to be aggressive do not chase after their targets with jars of hot sauce or headphones with which to administer bursts of white noise.” That small, r = .2 correlation I referenced before concerns behavior like that in a lab setting where experimental demand characteristics are almost surely present, suggesting the effect on aggressive behavior in naturalistic settings is likely overstated.)

Then again, in terms of meaningful impact, perhaps all those small effects weren’t really mounting to much. Indeed, the longitudinal research in this area seems to find the smallest effects (Anderson et al, 2010). To put that into what I think is a good example, imagine going to the gym. Listening to music helps many people work out, and the choice of music is relevant there. The type of music I would listen to when at the gym is not always the same kind I would listen to if I wanted to relax, or dance, or set a romantic mood. In fact, the music I listen to at the gym might even make me somewhat more aggressive in a manner of speaking (e.g., for an hour, aggressive thoughts might be more accessible to me while I listen than if I had no music, but that don’t actually lead to any meaningful changes in my violent behavior while at the gym or once I leave that anyone can observe). In that case, repeated exposure to this kind of aggressive music would not really make me any more aggressive in my day-to-day life than you’d expect overtime.

Thankfully, these warnings managed to save people from dangerous music

That’s not to say that media has no impact on people whatsoever: I fully suspect that people watching a horror movie probably feel more afraid than they otherwise would; I also suspect someone who just watched an action movie might have some violent fantasies in their head. However, I also suspect such changes are rather specific and of a short duration: watching that horror movie might increase someone’s fear of being eaten by zombies or ability to be startled, but not their fear of dying from the flu or their probability of being scared next week; that action movie might make someone think about attacking an enemy military base in the jungle with two machine guns, but it probably won’t increase their interest in kicking a puppy for fun, or lead to them fighting with their boss next month. These effects might push some feelings around in the very short term, but they’re not going to have lasting and general effects. As I said at the beginning of last week, things like violence are strategic acts, and it doesn’t seem plausible that violent media (like, say, comic books) will make them any more advisable.

References: Anderson, C. et al. (2010). Violent video game effects on aggression, empathy, and prosocial behavior in eastern and western counties: A meta-analytic review. Psychological Bulletin, 136, 151-173.

Bushman, B., Rothstein, H., & Anderson, C. (2010). Much ado about something: Violent video game effects and school of red herring: Reply to Ferguson & KIlburn (2010). Psychological Bulletin, 136, 182-187.

Elson, M. & Ferguson, C. (2013). Twenty-five years of research on violence in digital games and aggression: Empirical evidence, perspectives, and a debate gone astray. European Psychologist, 19, 33-46.

Ferguson, C. & Kilburn, J. (2010). Much ado about nothing: The misestimation and overinterpretation of violent video game effects in eastern and western nations: Comment on Anderson et al (2010). Psychological Bulletin, 136, 174-178.

Violence In Games Does Not Cause Real-Life Violence

Violence is a strategic act. What I mean by this is that a threat to employ physical aggression against someone else unless they do what you want is one that needs to be credible to be useful. If a 5-year-old child threatened to beat up her parents if they don’t stop for ice cream, the parents understand that the child does not actually pose a real physical risk and, if push came to shove, the parents would win a physical contest; by contrast, if you happen to be hanging out with a heavy-weight MMA fighter and he demands you pull over for ice cream, you should be more inclined to take his request seriously. If you cannot realistically threaten others with credible claims of violence – if you are not likely to be able to inflict harmful costs on others physically – then posturing aggressively shouldn’t be expected to do you any favors; if anything, adopting aggressive stances you cannot back up will result in your suffering costs inflicted by others, and that’s generally an outcome to be avoided. It’s for this reason that – on a theoretical level – we should have expected research on power poses to fail to replicate: simply acting more dominant will not make you more able to actually back up those boasts, and people shouldn’t be expected to take such posturing seriously. If you apply that same logic to nonhumans – say Rams – a male who behaves dominantly will occasionally encourage another male who will challenge that dominance. If neither backs down the result is a physical conflict, and the subsequent realization that writing metaphorical checks you cannot cash is a bad idea.

“You have his attention; sure hope you also have a thick skull, too”

This cursory analysis already suggests there might be a theoretical problem with the idea that people who are exposed to violent content in media will subsequently become more aggressive in real life. Yes, watching Rambo or playing Dark Souls might inspire some penchant for spilling fantasy blood (at least in the short term), but seeing violence doesn’t suddenly increase the advisability of your pursuing such a strategy, as you are no more likely to be able effectively employ it than you were before your exposure. Again, to place that in a nonhuman example (always a good idea when you’re dealing with psychology research to see if an idea still make sense; if it only makes sense for humans, odds are it’s lacking in interpretation), if you exposed a male ram to media depicting males aggressively slamming their horns into other males, that doesn’t suddenly mean your subject ram will be inspired to run out and challenge a rival. His chances of winning that contest haven’t changed, so why should his behavior?

Now the matter is more complex than this analysis lets on, admittedly, but it does give us something of a starting point for understanding why violent content in media – video games in particular – should not be expected to have uniform or lasting impacts on the player’s subsequent behavior. Before I get into the empirical side of this issue, however, I think it’s important I lay my potential bias on the table: I’m a gamer; have been my entire life, at least as far as I can remember. I’ve played games in all sorts of mediums – video, card, board, and sometimes combinations of those – and across a variety of genres, including violent ones. As such, when I see someone leveling accusations against one of my more cherished hobbies, my first response is probably defensive. That is, I don’t perceive people who research the link between violent games and aggression to be doing so for no particular reason; I assume they have some specific goals in mind (consciously understood or not) that center around telling other people what they shouldn’t do or enjoy, perhaps even ranging as far as trying to build a case for the censorship of such materials. As such, I’m by no means an unbiased observed in this matter, but I am also something of an expert in the subject matter as well, which can provide me with insights that others might not possess.

That disclaimer out the way, I wanted to examine some research today which examines the possibility that the relationship people have sometimes spotted between violent video game content and aggression isn’t casual (Przybylski et al, 2014; I say sometimes because apparently this link between the two is inconsistently present, possibly only short-term in nature, and the subject of some debate). The thrust of this paper focuses on the idea that human aggression (proximately) is a response to having one’s psychological needs thwarted. I think there are better ways to think about what aggression is, but this general idea is probably close enough to that truth to do us just fine. In brief, the idea motivating this paper is that people play video games (again, proximately), in part, because they provide feelings of competency and skill growth. Something about the challenges games offers to be overcome proves sufficiently motivating for players to get pleasure out of the experience. Importantly, this should hold true across gaming content: people don’t find content appealing because it is violent generally, but rather because it provides us abilities to test, display, and strengthen certain competencies. As such, manipulating the content of the games (from violent to non-violent) should be much less effective at causing subsequent aggression than manipulating the difficulty of the game (from easy/intuitive to difficult/confusing).    

“I’ll teach him a lesson about beating me in Halo”

This is a rather important factor to consider because the content of a game (whether it is violent or not, for instance) might be related to how difficult the game is to learn or master. As such, if researchers have been trying to vary the content without paying much mind to the other factors that correlate with it, that could handicap the usefulness of subsequent interpretations. Towards that end, Przybylski et al (2014) report on the results of seven studies designed to examine just that issue. I won’t be able to go over all of them in depth, but try to provide a general adequate summary of their methods and findings. In their first study, they examined how 99 participants reacted to playing a simple but non-violent game (about flying a paper airplane through rings) or a complex but violent one (a shooter with extensive controls). The players were then asked about their change in aggressive feelings (pre- and post-test difference) and mastery of the controls. The quick summary of the results was that aggressive content did not predict change in aggression scores above and beyond the effects of frustrations over the controls, while the control scores did predict aggression.

Their second study actually manipulated the content and complexity factors (N = 101). Two versions of the same game (Half-Life 2) were created, such that one contained violent content and the other did not, while the overall environment and difficulty were held constant. Again, there were no effects of content on aggression, but there was an effect of perceived mastery. In other words, people felt angry when they were frustrated with the game; not because of the content. Their third study (N = 104) examined what happened when a non-violent puzzle game (Tetris) was modified to either contain simple or complex control interface. As before, those who had to deal with the frustrating controls were quicker to access aggressive thoughts and terms than those in the intuitive control condition. Study 4 basically repeated that design with some additional variables and found the same type of results: perceived competency in the game correlated negatively with aggression and that people become more aggressive the less they enjoyed the game, among a few other things.The fifth study had 112 participants all play a complex game that was either (a) violent or non-violent, but also gave them either (b) 10 minutes of practice time with the game or no experience with it. As expected, there was an effect of being able to practice on subsequent aggression, but no effect of violent content.

Study 6 asked participants to first submerge their arm in ice water for 25 seconds (a time period ostensibly determined by the last participant), then play a game of Tetris for a few minutes that was modified to be either easy or difficult (but not because of the controls this time). Those assigned to play the more difficult version of Tetris also reported more aggressive feelings, and assigned the next subject to submerge their arm for about 30 seconds in the ice water (relative to the 22 second average assignment in the easy group). The final study surveyed regular players about their experiences gaming over the last month and aggressive feelings, again finding that the ratings of content did not predict aggressive self-reported reactions to gaming, but frustrations with playing the game did.

“I’m going to find the developer of this game and kill him for it!”

In summation, then, violent content per se does not appear to make players more aggressive; instead, frustration and losing seem to play a much larger role. It is at this point that my experience as a gamer comes in handy, because such an insight should be readily apparent to anyone who has watched many other people play games. As an ever-expanding library of YouTube rage-quit videos document, a gamer can become immediately enraged by losing at almost any game, regardless of the content (for those of you not in the know, rage-quitting refers to aggressively quitting out of a game following a loss, often accompanied by yelling, frustrating, and broken controllers). I’ve seen people losing their minds over shooters, sports games, card games, board games, and storming off while shouting. Usually such outbursts are short-term affairs – you don’t see that person the next day and notice they’re visibly more aggressive towards others indiscriminately – but the important part is that they almost always occur in response to losses (and usually losses deemed to be unfair, in some sense).

As a final thought, in addition to the introductory analysis and empirical evidence presented here, there are other reasons one might not predict that violent content per se would be related to subsequent aggression even if one wants to hold onto the idea that mere exposure to content is enough to alter future behavior. In this case, most of the themes found within games that have violent content are not violence and aggression as usually envisioned (like The Onion‘s piece on Close Range: the video game about shooting people point blank in the face). Instead, those themes usually focus on the context in which that violence is used: defeating monsters or enemies that threaten the safety of you or others, killing corrupt and dangerous individuals in positions of power, or getting revenge for past wrongs. Those themes are centered more around heroism and self-defense than aggression for the sake of violence. Despite that, I haven’t heard of many research projects examining whether playing such violent games could lead to increased psychological desires to be helpful, or encourage people to take risks to save others from suffering costs.

References: Przybylski, A., Rigby, C., Deci, E., & Ryan, R. (2014). Competent-impeding electronic games and players’ aggressive feelings, thoughts, and behaviors. Journal of Personality & Social Psychology, 16, 441-457.

Sensitive Topics: Not All That Sensitive

Standards and Practices are a vital link in keeping good and funny ideas away from you, the television viewer

If you’ve ever been involved in getting an academic research project off the ground, you likely share some form of frustration with the Institutional Review Boards (or IRBs) that you had to go through before you could begin. For those of you not the know, the IRB is an independent council set up by universities tasked with assessing and monitoring research proposals associated with the university for possible ethical violations. Their main goal is in protecting subjects – usually humans, but also nonhumans – from researchers who might otherwise cause them harm during the course of research. For instance, let’s say a researcher is testing an experimental drug for effectiveness in treating a harmful illness. The research begins by creating two groups of participants: one who receive the real drug and one who receives a placebo. Over the course of the study, if it becomes apparent that the experimental drug is working, it would be considered unethical for the researcher to withhold the effective treatment from the placebo group. Unfortunately, ethical breaches like that have happened historically and (probably) continue to happen today. It’s the IRB’s job to help reduce the prevalence of such issues.

Because the research ethics penguin just wasn’t cutting it

Well-intentioned as the idea is, the introduction of required IRB approval to conduct any research involving humans – including giving them simple surveys to fill out – places some important roadblocks in the way of researcher efficiency; in much the same way, after the 9/11 attacks airport security became much more of a headache to get through. First and foremost, the IRB usually requires a lot of paperwork and time for the proposal to be processes and examined. It’s not all that unusual for what should be a straightforward and perfectly ethical research project to sit in the waiting room of the IRB for six-to-eight weeks just to get green lit. That approval is not always forthcoming, though, with the IRB sending back revisions or concerns about projects regularly; revisions which, in turn, can hold the process up for additional days or weeks. For any motivated researcher, these kinds of delays can be productivity poison, as one’s motivation to conduct a project might have waned somewhat over the course of the two or three months since its inception. If you’re on a tight deadline, things can get even worse.

On the subject of concerns the IRB might express over research, today I wanted to talk about a matter referred to as sensitive topics research. Specifically, there are some topics – such as those related to sexual behavior, trauma, and victimization – that are deemed to pose greater than minimal risk to participants being asked about them. The fear in this case stems from the assumption that merely asking people (usually undergraduates) about these topics could be enough to re-traumatize them and cause them psychological distress above and beyond what they would experience in daily life. In that sense, then, research on certain topics can deemed above minimal risk, resulting in such projects being put under greater scrutiny and ultimately subjected to additional delays or modifications (relative to more “low-risk” topics like visual search tasks or personality measures, anyway).

That said, the IRBs are not necessarily composed of experts on the matter of ethics, nor do their concerns need empirical grounding to be raised; the mere possibility that harm might be caused can be considered grounds enough for not taking any chances and risking reputational or financial damage to the institution (or the participants, of course). That these concerns were raised frequently (but not supported) led Yeater et al (2012) to examine the matter empirically. The authors sought to subject their participants to a battery of questions and measures designated to be either (a) minimal risk, which were predominately cognitive tasks, or (b) above minimal risk, which were measures that asked about matters like sexual behavior and trauma. Before and after each set of measures, the participants would have their emotional states measured to see if any negative or positive changes resulted from taking part in the research.

The usual emotional response to lengthy surveys is always positive

The sample for this research involved approximately 500 undergraduates assigned to either the trauma-sex condition (n = 263) or the cognitive condition (n = 241). All of the participants first completed some demographic and affect measures designed to assess their positive and negative emotions. After that, those in the trauma-sex condition filled out surveys concerning their dating behavior, sexual histories, the rape myth acceptance scale, questions concerning their interest in short-term sex, sexual confidence, trauma and post-traumatic checklists, and childhood sexual and trauma histories. Additionally, females answered questions about their body, menstrual cycle, and sexual victimization histories; males completed similar surveys asking about their bodies, masturbation schedules, and whether they had sexually victimized women. Those in the cognitive condition filled out a similarly-long battery of tests measuring things like their verbal and abstract reasoning abilities.

Once these measures were completed, the emotional state of all the participants was again assessed along with other post-test reaction questions, including matters like whether they perceived any costs and benefits from engaging in the study, how mentally taxing their participation felt, and how their participation measured up to other life stressors in life like losing $20, getting a paper cut, a bad grade on a test, or waiting on line in the bank for 20 minutes.

The results from the study cut against the idea that undergraduate participants were particularly psychologically vulnerable to these sensitive topics. In both conditions, participants reported a decrease in negative affect over the course of the study. There was even an increase in positive affect, but only for the trauma-sex group. While those in the trauma-sex condition did report greater post-test negative emotions, the absolute value of those negative emotions were close to floor levels for both groups (both means were below a 2 on a scale of 1-7). That said, those in the trauma-sex condition also reported lower mental costs to taking part in the research and perceived greater benefits overall. Both groups reported equivalent positive emotions.

Some outliers were then considered. In terms of those reporting negative emotions, 2.1% of those in the cognitive condition (5 participants) and 3.4% of those in the trauma-sex condition (9 participants) reported negative emotions above the midpoint of the scale. However, the maximum value for those handful of participants were 4.15 and 5.52 (respectively) out of 7, falling well short of the ceiling. Looking specifically at women who had reported histories of victimization, there was no apparent difference between conditions with regard to affect on almost any of the post-test measures; the one exception was that women who had experienced a history of victimization reported the trauma-sex measures to be slightly more mentally taxing, but that could be a function of their having to spend additional time filling out the large number of extensive questionnaires rather than any kind of serious emotional harm. Even those who had been harmed in the past didn’t seem terribly bothered by answering some questions.

“While we have you here, would you like to answer a quick survey about your experience?”

The good news is that it would seem undergraduates are more resilient than they are often given credit for and not so easily triggered by topics like sex or abuse (which are frequently discussed on social platforms like Facebook and news sources). The sensitive topics didn’t seem to be all that sensitive; certainly not substantially more so than the standard types of minimal risk questions asked on other psychological measures. Even for those with histories of victimization. The question remains as to whether such a finding would be enough to convince those making the decisions about the risks inherent in this kind of research. I’d like to be optimistic on that front, but it would rely on the researchers being aware of the present paper (as you can’t rely on the IRB to follow the literature on that front, or indeed any front) and the IRB being open to hearing evidence to the contrary. As I have encountered reviewers who seem uninterested in hearing contrary evidence concerning deception, it’s a distinct possibility that the present research might not have the intended effect on mollifying IRB concerns. I certainly wouldn’t rule out it’s potential effectiveness, though, and this is definitely a good resource for researchers to have in their pocket if they encounter such issues.

References: Yeater, E., Miller, G., Rinehart, J., & Nason, E. (2012). Trauma and sex surveys meet minimal risk standards: Implications for institutional review boards. Psychological Science, 23, 780-787.

 

Spinning Sexism Research On Accuracy

When it comes to research on sexism, there appear to be many parties interested in the notion that sexism ought to be reduced. This is a laudable goal, and one that I would support; I am very much in favor in treating people as individuals rather than representatives of their race, sex, or any other demographic characteristics. It is unfortunately, however, that this goal often gets side-tracked by an entirely different one: trying to get people to reduce the extent to which people view men and women as different. What I mean by this is that I have seen many attempts to combat sexism by trying to reduce the perception that men and women differ in terms of their psychology, personality, intelligence, and so on; it’s much more seldom that those same voices appear to convince people who inaccurately perceive sex differences as unusually small to adjust their estimate upwards. In other words, rather that championing accuracy is perceptions, there appears to be a more targeted effort for minimizing particular differences; while those are sometimes the same thing (sometimes people are wrong because they overestimate), they are often not (sometimes people are wrong because they underestimate), and when those goals do overlap, the minimization side tends to win out.

Just toss your perceptions in with the rest of the laundry; they’ll shrink

In my last post, I discussed some research by Zell et al (2016) primarily in the service of examining measures of sexism and the interpretation of the data they produce (which I recommend reading first). Today I wanted to give that paper a more in-depth look to illustrate this (perhaps unconscious) goal of trying to get people to view the sexes as more similar than they actually are. Zell et al (2016) begin their introduction by suggesting that most psychological differences between men and women are small, and the cases in which medium to large differences exist – like mating preferences and aggression – tend to be rare. David Schmitt has already put remarks like that into some context, and I highly recommend you read his post on the subject. In the event you can’t be bothered to do so at the moment, one of the most important takeaway points from his post is that even if the differences in any one domain tend to be small on average, when considered across all those domains simultaneously, those small differences can aggregate into much larger ones.

Moreover, the significance of a gender difference is not necessarily determined by its absolute size, either. This was a point Steven Pinker mentioned in a somewhat-recent debate with Elizabeth Spelke (and was touched on again in a recent talk by Jon Haidt at SUNY New Paltz). To summarize this point briefly, if you’re looking at a trait in two normally-distributed populations that are, on average, quite similar, the further from that average value you get, the most extreme the difference between populations become. Pinker makes the point clear in this example:

“…it’s obvious that distributions of height for men and women overlap: it’s not the case that all men are taller than all women. But while at five foot ten there are thirty men for every woman, at six feet there are two thousand men for every woman. Now, sex differences in cognition tend not to be so extreme, but the statistical phenomenon is the same.”

Not only are small sex differences sometimes important, then, (such as when you’re trying to hire people for a job who are in the top 1% of distribution for a trait like intelligence, speed, conscientiousness; you name it) but a large number of small effects (as well as some medium and large ones) can all add up to collectively represent some rather large differences (and that assumes you’re accounting for all relevant sex differences; not just a non-representative sample of them). With all this considered, the declaration at the beginning of Zell et al’s paper that most sex differences tend to be small strikes me less as a statement of empirical concern, but rather one that serves to set up the premise for the rest of their project: specifically, the researchers wanted to test whether people’s scores on the ambivalent sexism inventory predicted (a) the extent to which they perceive sex differences as being large and (b) the extent to which they are inaccurate in their perceptions. The prediction in this case was that people who scored high on their ostensible measures of sexism would be more likely to exaggerate sex differences and more likely to be wrong about their size overall (as an aside, I don’t think those sexism questions measure what the authors hope they do; see my last post).

Pictured: Something not even close to what was being assessed in this study

In their first study, Zell et al (2016) asked about 320 participants to estimate how large they think sex differences are between men and women (from 1-99) were for 48 traits and to answer 6 questions intended to measure their hostile and benevolent sexism (as another aside, I have no idea why those 48 traits in particular were selected). These answers were then averaged for each participant to create an overall score for how large they viewed the sex differences to be, and how high they scored on hostile and benevolent sexism. When the relevant factors were plugged into their regression, the results showed that those higher in hostile (ß = .19) and benevolent (ß = .29) sexism tended to perceive sex differences as larger, on average. When examined by gender, it was found that women (ß = .41) who were higher in benevolent sexism were more likely to perceive sex differences as large (but this was not true for men: ß = .11) and – though it was not significant – the reverse pattern held for hostile sexism, such that women high in hostile sexism were nominally less likely to perceive sex differences as large (ß = -.32).

The more interesting finding, at least as far as I’m concerned, is that in spite of those scoring higher on their sexism scores perceiving sex differences to be larger, they were not really more likely to be wrong about them. Specifically, those who scored higher on benevolent sexism were slightly less accurate (ß = -.20), just as women tended to be less accurate than men (ß = -.19); however, hostile sexism scores were unrelated to accuracy altogether (ß = .003), and no interactions with gender and sexism emerged. To put that in terms of the simple correlations, hostile and benevolent sexism correlated much better with the perceived size of sex differences (rs = .26 and .43, respectively) than they did with accuracy (rs = -.12 and -.22, with the former not being significant and the latter being rather small). Now since we’re dealing with two genders, two sexism scales, and relatively small effects, it is possible that some of these findings are a bit more likely to be statistical flukes; that does tend to happen as you keep slicing data up. Nevertheless, these results are discussed repeated within the context of their paper as representing exaggerations: those scoring higher on these sexism measures are said to exaggerate sex differences, which is odd on account of them not consistently getting them all that wrong.

This interpretation extends to their second study as well. In that experiment, about 230 participants were presented with two mock abstracts and told that only one of them represented an accurate summary of psychological research on sex differences. The accurate version, of course, was the one that said sex differences were small on average and therefore concluded that men and women are very similar to each other, whereas the bogus abstract concluded that gender differences are often large and therefore men and women are very different from one another. As I reviewed in the beginning of the post, small differences can often have meaningful impacts both individually and collectively, so the lines about how men and women are very similar to each other might not reflect an entirely accurate reading of the literature even if the part about small average sex differences did. This setup is already conflating the two statements (“average effect sizes on all these traits is small” and “men and women are very similar across the board”).

“Most of the components aren’t that different from modern cars, so they’re basically the same”

As before, those higher in hostile and benevolent sexism tended to say that the larger sex difference abstract more closely reflected their personal views (women tended to select the large-difference abstract 50.4% of the time compared to men’s 44.2% as well). Now because the authors view the large sex difference abstract as being the fabricated one, they conclude that those higher in those sexism measures are less accurate and more likely to exaggerate these views (they also make a remark that their sexism measures indicate which people “endorse sexist ideologies”; a determination it’s not at all cut out for making). In other words, the authors interpret this finding as those selecting the large-differences abstract to hold “empirically unsupported” views (which in a sort-of ironic sense means that, as the late George Carlin put it, “Men are better at it” when it comes to recognizing sex differences).

This is an interesting methodological trick they employ: since they failed to find much in the way of a correlation between sexism scores and accuracy in their first study (it existed sometimes, but was quite small across the board and certainly much smaller than the perception of size correlation), they created a coarser and altogether worse measure of accuracy in the second study and use that to support their views that believing men and women tend to be rather different is wrong instead. As the old saying goes, if at first you don’t succeed, change your measures until you do.

References: Zell, E., Strickhouser, J., Lane, T., & Teeter, S. (2016). Mars, Venus, or Earth? Sexism and the exaggeration of psychological gender differences. Sex Roles, 75, 287-300.