Research Tip: Ask About What You Want To Measure

Recently I served as a reviewer for a research article that had been submitted to a journal for publication. Without going into too much detail as to why, the authors of this paper wanted to control for people’s attitudes towards casual sex when conducting their analysis. They thought that it was possible people who were more sexually-permissive when it comes to infidelity might respond to certain scenarios differently than those who were less sexually-permissive. If you were the sensible type of researcher, you might do something like ask your participants to indicate on some scale as to how acceptable or unacceptable they think sexually infidelity is, then. The authors of this particular paper opted for a different, altogether stranger route: they noted that people’s attitudes towards infidelity correlate (imperfectly) with their political ideology (i.e., whether they consider themselves to be liberals or conservatives). So, rather than ask participants directly about how acceptable infidelity is (what they actually wanted to know), they asked participants about their political ideology and used that as a control instead.

 ”People who exercise get tired, so we measured how much people napped to assess physical fitness”

This example is by no means unique; psychology researchers frequently try to ask questions about topic X in the hopes of understanding something about topic Y. This can be acceptable at times, specifically when topic Y is unusually difficult – but not impossible – to study directly. After all, if topic Y is impossible to directly study, then one obviously cannot say that studying topic X tells you something about Y with much confidence, as you would have no way of assessing the relationship between X and Y to begin with. Assuming that the relationship between X and Y has been established and it is sufficiently strong and Y is unusually difficult to study directly, then there’s a good, practical case to be made for using X instead. When that is done, however, it should always be remembered that you aren’t actually studying what you’d like to study, so it’s important to not get carried away with the interpretation of your results.

This brings us nicely to the topic of research on sexism. When people hear the word “sexism” a couple things come to mind: someone who believes one sex is (or should be) – socially, morally, legally, psychologically, etc – inferior to the other, or worth less; someone who wouldn’t want to hire a member of one sex for a job (or intentionally pays them less if they did) strictly because of that variable regardless of their qualifications; someone who inherently dislikes members of one sex. While this list is by no means exhaustive, I suspect things like these are probably the prototypical examples of sexism; some kind of explicit, negative attitude about people because of their sex per se that directly translates into behavior. Despite this, people who research sexism don’t usually ask about such matters directly, as far as I’ve seen. To be clear, they easily could ask such questions assessing such attitudes in straightforward manners (in fact, they used to do just that with measures like the “Attitudes Towards Women Scale” in the 1970s), but they do not. As I understand it, the justification for not asking about such matters directly is because it has become more difficult to find people who actually express such views (Loo & Thorpe, 1998). As attitudes had already become markedly less sexist from 1972 to 1998, one can only guess at how much more change occurred from then to now. In short, it’s becoming rare to find blatant sexists anymore, especially if you’re asking college students.

Many researchers interpret that difficulty as being the result of people still holding sexist attitudes but either (a) are not willing express them publicly for fear of condemnation, or (b) are not consciously aware that they hold such views. As such, researchers like to ask about questions about “Modern Sexism” or “Ambivalent Sexism“; they maintain the word “sexism” in their scales, but they begin to ask about things which are not what people first think of when they hear the term. They no longer ask about explicitly sexist attitudes. Therein lies something of a problem, though: if what you really want to know is whether people hold particular sexist beliefs or attitudes, you need some way of assessing those attitudes directly in order to determine that other questions which don’t directly ask about that sexism will accurately reflect it. However, if such a method of assessing those beliefs accurately, directly, and easily does exist, then it seems altogether preferable to use that method instead. In short, just ask about the things you want to ask about. 

“We wanted to measure sugar content, so we assessed how much fruit the recipe called for”

If you continue on with using an alternate measure – like using the Ambivalent Sexism Inventory (ASI), rather than the Attitudes towards Women Scale – then you really should restrict your interpretations to things you’re actually asking about. As a quick example, let’s consider the ASI, which is made up of a hostile and benevolent sexism component. Zell et al (2016) summarize the scale as follows:

“Hostile sexism is an adversarial view of gender relations in which women are perceived as seeking control over men. Benevolent sexism is a subjectively positive view of gender relations in which women are perceived as pure creatures who ought to be protected, supported, and adored; as necessary companions to make a man complete; but as weak and therefore best relegated to traditional gender roles (e.g., homemaker).”

In other words, the benevolent scale measures the extent to which women are viewed as children: incapable of making their own decisions and, as such, in need of protection and provisioning by men. The hostile scale measures the extent to which men don’t trust women and view them as enemies. Glick & Fiske (1996) claim that  ”...hostile and benevolent sexism…combine notions of the exploited group’s lack of competence to exercise structural power with self-serving “benevolent” justifications.” However, not a single measure on either the hostile or benevolent sexism inventory actually asks about female competencies or whether women ought to be restricted socially. 

To make this explicit, let’s consider the questions Zell et al (2016) used to assess both components. In terms of hostile sexism, participants were asked to indicate their agreement with the following three statements:

  • Women seek power by gaining control over men
  • Women seek special favors under the guise of equality
  • Women exaggerate their problems at work

There are a few points to make about these questions: first, they are all clearly true to some extent. I say that because these are behaviors that all kinds of people engage in. If these behaviors are not specific to one sex – if both men and women exaggerate their problems at work – then agreement with the idea that women do does not stop me from believing men do this as well and, accordingly, does not necessarily track any kind of sexist belief (the alternative, I suppose, is to believe that women never exaggerate problems, which seems unlikely). If the questions are meant to be interpreted as a relative statement (e.g., “women exaggerate their problems at work more than men do”), then that statement needs to first be assessed empirically as true or false before you can say that endorsement of it represents sexism. If women actually do tend to exaggerate problems at work more (a matter that is quite difficult to objectively determine because of what the term exaggerate means), then agreement with the statement just means you accurately perceive reality; not that you’re a sexist.

More to the point, however, none of the measures ask about what the researchers interpret them to mean: women seeking special favors does not imply they are incompetent or unfit to hold positions outside of the home, nor does it imply that one views gender relations primarily as adversarial. If those views are really what a researcher is trying to get at, then they ought to just ask about them directly. A similar story emerges for the benevolent questions:

  • Women have a quality of purity few men possess
  • Men should sacrifice to provide for women
  • Despite accomplishment, men are incomplete without women

 Again, I see no mention of women’s competency, ability, intelligence, or someone’s endorsement of strict gender roles. Saying that men ought to behave altruistically towards women in no way implies that women can’t manage without men’s help. When a man offers to pay for an anniversary dinner (a behavior which I have seen labeled sexist before), he is usually not doing so because he feels his partner is incapable of paying anymore than my helping a friend move suggests I view them as a helpless child. 

“Our saving you from this fire implies you’re unfit to hold public office”

The argument can, of course, be made that scores on the ASI are related to the things these researchers actually want to measure. Indeed, Glick & Fiske (1996) made that very argument: they report that the hostile sexism scores (controlling for the benevolent scores) did correlate with “Old Fashion Sexism” and “Attitudes towards Women” scores (rs = .43 and .60, respectively, bearing in mind that was almost 20 years ago and these attitudes are changing). However, the correlations between benevolent sexism scores and these sexist attitudes were effectively zero (rs = -.03 and .04, respectively). In other words, it appears that people endorse these statements for reasons that have nothing at all to do with whether they view women as weak, or stupid, or any other pejorative you might throw out there, and their responses may tell you nothing at all about their opinion concerning gender roles. If you want to know about those matters, then ask about them. In general, it’s fine to speculate about what your results might mean – how they can best be interpreted – but an altogether easier path is to simply ask about such matters directly and reduce the need for pointless speculation.

 References: Glick, P. & Fiske, S. (1996). The ambivalent sexism inventory: Differentiating hostile and benevolent sexism. Journal of Personality & Social Psychology, 70, 491-512.

Loo, R. & Thorpe, K. (1998). Attitudes towards women’s roles in society: A replication after 20 years. Sex Roles, 39, 903-912.

Zell, E., Strickhouser, J., Lane, T., & Teeter, S. (2016). Mars, Venus, or Earth? Sexism and the exaggeration of psychological gender differences. Sex Roles, 75, 287-300.

The Value Of Association Value

Sometime ago I was invited to give a radio interview regarding a post I had written: The Politics of Fear. Having never been exposed to this kind of a format before, I found myself having to try and make some adjustments to my planned presentation on the fly, as it quickly became apparent that the interviewer was looking more for quick and overly-simplified answers, rather than anything with real depth (and who can blame him? It’s not like many people are tuning into the radio with the expectation of receiving anything resembling a college education). At one point I was posed with a question along the lines of, “how people can avoid letting their political biases get the better of them,” which was a matter I was not adequately prepared to answer. In the interests of compromise and giving the poor host at least something he could work with (rather than the real answer: “I have no idea; give me a day or two and I’ll see what I can find”), I came up with a plausible sounding guess: try to avoid social isolation of your viewpoints. In other words, don’t remove people from your friend groups or social media just because you disagree with they they say, and actively seek out opposing views. I also suggested that one attempt to expand their legitimate interests in the welfare of other groups in order to help take their views more seriously. Without real and constant challenges to your views, you can end up stuck in a political and social echo chamber, and that will often hinder your ability to see the world as it actually is.

“Can you believe those nuts who think flooding poses real risks?”

As luck would have it, a new paper (Almaatouq et al, 2016) fell into my lap recently that – at least in some, indirect extent – helps speak to the quality of the answer I had provided at the time (spoiler: as expected, my answer was pointing in the right direction but was incomplete and overly-simplified). The first part of the paper examines the shape of friendships themselves: specifically whether they tend to be reciprocal or more unrequited in one direction or the other. The second part leverages those factors to try and explain what kinds of friendships can be useful for generating behavioral change (in this case, getting people to be more active). Put simply, if you want to change someone’s behavior (or, presumably, their opinions) does it matter if (a) you think they’re your friend, but they disagree, (b) they think you’re their friend, but you disagree, (c) whether you both agree, and (d) how close you are as friends?

The first set of data reports on some general friendship demographics. Surveys were provided to 84 students in a single undergraduate course that asked to indicate, from 0-5, whether they considered the other students to be strangers (0), friends (3), or one of their best friends (5). The students were also asked to predict how each other student in the class would rate them.  In other words, you would be asked, “How close do you rate your relationship with X?” and “How close does X rate their relationship to you?” A friendship was considered mutual if both parties rated each other as at least a 3 or greater. There was indeed a positive correlation between the two ratings (r = .36), as we should expect: if I rate you highly as a friend, there should be a good chance you also rate me highly. However, that reality did diverge significantly from what the students predicted. If a student has nominated someone as a friend, their prediction as to how that person would rate them showed substantially more correspondence (r = .95). Expressed in percentages, if I nominated someone as a friend, I would expect them to nominate me back about 95% of the time. In reality, however, they would only do so about 53% of the time.

The matter of why this inaccuracy exists is curious. Almaatouq et al, (2016) put forward two explanations, one of which is terrible and one of which is quite plausible. The former explanation (which isn’t really examined in any detail, and so might just have been tossed in) is that people are inaccurate at predicting these friendships because non-reciprocal friendships “challenge one’s self-image.” This is a bad explanation because (a) the idea of a “self” isn’t consistent with what we know about how the brain works, (b) maintaining a positive attitude about oneself does nothing adaptive per se, and (c) it would need to posit a mind that is troubled by unflattering information and so chooses to ignore it, rather than the simpler solution of a mind that is simply not troubled by such information in the first place. The second, plausible explanation is that some of these ratings of friendships actually reflect some degree of aspiration, rather than just current reality: because people want friendships with particular others, they behave in ways likely to help them obtain such friendships (such as by nominating their relationship as mutual). If these ratings are partially reflective of one’s intent to develop them over time, that could explain some inaccuracy.

Though not discussed in the paper, it is also possible that perceivers aren’t entirely accurate because people intentionally conceal friendship information from others. Imagine, for instance, what consequences might arise for someone who finally works up the nerve to go tell their co-workers how they really feel about them. By disguising the strength of our friendships publicly, we can leverage social advantages from that information asymmetry. Better to have people think you like them than know you don’t in many cases.

 ”Of course I wasn’t thinking of murdering you to finally get some quiet”

With this understanding of how and why relationships can be reciprocal or asymmetrical, we can turn to the matter of how they might influence our behavior and, in turn, how satisfactory my answer was. The authors utilized a data set from the Friends and Family study, which had asked a group of 108 people to rate each other as friends on a 0-7 scale, as well as collected information about their physical activity level (passively, via a device in their smartphones). In this study, participants could earn money by becoming more physically active. In the control condition, participants could only see their own information; in the two social conditions (that were combined for analysis) they could see both their own activity levels and those of two other peers: in one case, participants earned a reward based only on their own behavior, and in the other the reward was based on the behavior of their peers (it was intended to be a peer-pressure condition). The relationship variables and conditions were entered into a regression to predict the participant’s change in physical activity.

In general, having information about the activity levels of peers tended to increase the activity of the participants, but the nature of those relationships mattered. Information about the behavior of peers in reciprocal friendships had the largest effect (b = 0.44) on affecting change. In other words, if you got information about people you liked who also liked you, this appeared to be most relevant. The other type of relationship that significantly predicted change was one in which someone else valued you as a friend, even if you might not value them as much (b = 0.31). By contrast, if you valued someone else who did not share that feeling, information about their activity didn’t seem to predict behavioral changes well (b = 0.15) and, moreover, the strength of friendships seemed to be rather besides the point (b = -0.04), which was rather interesting. Whether people were friends seemed to matter more than the depth of that friendship.

So what do these results tell us about my initial answer regarding how to avoid perceptual biases in the social world? This requires a bit of speculation, but I was heading in the right direction: if you want to affect some kind of behavioral change (in this case, reducing one’s biases rather than increasing physical activity), information from or about other people is likely a tool that could be effectively leveraged for that end. Learning that other people hold different views than your own could cause you to think about the matter a little more deeply, or in a new light. However, it’s often not going to be good enough to simply see these dissenting opinions in your everyday life if you want to end up with a meaningful change. If you don’t value someone else as an associate, they don’t value you, or neither of you value the other, then their opinions are going to be less effective at changing yours than they otherwise might be, relative to when you both value each other.

At least if mutual friendship doesn’t work, there’s always violence

The real tricky part of that equation is how one goes about generating those bonds with others who hold divergent opinions. It’s certainly not the easiest thing in the world to form meaningful, mutual friendships with people who disagree (sometimes vehemently) with your outlooks on life. Moreover, achieving an outcome like “reducing cognitive biases” isn’t even always an adaptive thing to do; if it were, it would be remarkable that those biases existed in the first place. When people are biased in their assessment of research evidence, for instance, they’re usually biased because something is on the line, as far as they’re concerned. It does an academic who has built his career on his personal theory no favors to proudly proclaim, “I’ve spent the last 20 years of my life being wrong and achieving nothing of lasting importance, but thanks for the salary and grant funding.” As such, the motivation to make meaningful friendships with those who disagree with them is probably a bit on the negative side (unless their hope is that through this friendship they can persuade the other person to adopt their views, rather than vice versa because – surely – the bias lies with other people; not me). As such, I’m not hopeful that my recommendation would play out well in practice, but at least it sounds plausible enough in theory.

References: Almaatouq, A., Radaelli, L., Pentland, A., & Shmueli, E. (2016). Are you your friends’ friends? Poor perception of friendship ties limits the ability to promote behavioral change. PLOS One, 11, e0151588. doi:10.1371/journal.pone.0151588