Monthly Archives: December 2014

An Eye For Talent

Posted on by

Rejection can be a painful process for almost anyone (unless you’re English). For many, rejection is what happens when a (perhaps overly-bloated) ego ends up facing the reality that it really isn’t as good as it likes to tell people it is. For others, rejection is what happens when the person in charge of making the decision doesn’t possess the accuracy of assessment that they think they do (or wish they did), and failed to recognize your genius. One of the most notable examples of the latter is The Beatle’s Decca audition in 1962, during which the band was told they had no future in show business. Well over 250 million certified sales later, “oops” kind of fails to cut it with respect to how large of a blunder that decision was. This is by no means a phenomenon unique to The Beatles either: plenty of notable celebrities had been previously discouraged or rejected from their eventual profession by others. So we have a bit of error management going on here: record labels want to do things like (a) avoid signing artists that are unlikely to go anywhere while (b) avoiding failures to sign the best-selling band of all time. As they can’t do either of those things with perfect accuracy, they’re bound to make some mistakes.

“Yet again, our talents have gone unnoticed despite our sick riffs”

Part of the problem facing companies that put out products such as albums, books, movies, and the rest, is that popularity can be a terribly finicky thing, since popularity can often snowball on itself. It’s not necessarily the objective properties of a song or book that make it popular; a healthy portion of popularity depends on who else likes it (which might sound circular, but it’s not). This tends to make the former problem of weeding out the bad artists easier than finding the superstars: in most cases, people who can’t sing well won’t sell, but just because one can sing well it doesn’t mean they’re going to be a hit. As we’re about to see, these problems are shared not only by people who put out products like music or movies; they’re also shared by people who publish (or fail to publish) scientific research. A recent paper by Siler, Lee, & Bero (2014) sought to examine how good the peer review process – the process through which journal editors and reviewers decide what gets published and what does not – is at catching good papers and filtering out bad ones.

The data examined by the authors focused on approximately 1,000 papers that had been submitted to three of the top medical journals between 2003 and 2004: Annals of Internal Medicine, British Medical Journal, and The Lancet. Of the 1,008 manuscripts, 946 – or about 94% of them – were rejected. The vast majority of those rejections – about 80% – were desk rejections, which is when an article is not sent out for review before the journal decides to not publish it. From that statistic alone, we can already see that these journals are getting way more submissions than they could conceivably publish or review and, accordingly, lots of people are going to be unhappy with their decision letters. Thankfully, publication isn’t a one-time effort; authors can, and frequently do, resubmit their papers to other journals for publication. In fact, 757 of the rejected papers were found to have been subsequently published in other journals (more might have been published after being modified substantially, which would make them more difficult to track). This allowed Siler, Lee, & Bero (2014) the opportunity to compare the articles that were accepted to those which were rejected in terms of their quality and importance.

Now determining an article’s importance is a rather subjective task, so the authors decided to focus instead on the paper’s citation counts – how often other papers had referenced them – as of April 2014. While by no means a perfect metric, it’s a certainly a reasonable one, as most citations tend to be positive in nature. First, let’s consider the rejected articles. Of the articles that had been desk rejected by one of the three major journals but eventually published in other outlets, the average citation count was 69.8 per article; somewhat lower than the articles which had been sent out for review before they had been rejected (M = 94.65). This overstates the “average” difference by a bit, however, as citation count is not distributed normally. In the academic world, some superstar papers receive hundreds or thousands of the citations, whereas many others hardly receive any. To help account for this, the authors also examined the log-transformed number of citations. When they did so, the mean citation count for the desk rejected papers was 3.44, and 3.92 for the reviewed-then-rejected ones. So that is some evidence consistent with the notion that those who decide whether or not to send papers out for review work as advertised: the less popular papers (which we’re using as a proxy for quality) were rejected more readily, on average.

“I just don’t think they’re room for you on the team this season…”

There’s also evidence that, if the paper gets sent out to reviewers, the peer reviewers are able to assess a paper’s quality with some accuracy. When reviewers send their reviews back to the journal, they suggest that the paper be published as is, with minor/major revisions, or rejected. If those suggestions are coded as numerical values, each paper’s mean reviewer score can be calculated (e.g., fewer recommendations to reject = better paper). As it turns out, these scores correlated weakly – but positively – with an article’s subsequent citation count (r = 0.28 and 0.21 with citation and logged citation counts, respectively), so it seems the reviewers have at least some grasp on the paper’s importance and quality as well. That said, the number of times an article was revised prior to acceptance had no noticeable effect on it’s citation count. While reviewers might be able to discern the good papers from the bad at better-than-chance rates, the revisions they suggested did not appear to have a noticeable impact on later popularity.

What about the lucky papers that managed to get accepted by these prestigious journals? As they had all gone out for peer review, the reviewer’s scores were again compared against citation count, revealing a similarly small but positive correlation (0.21 and 0.26 with citation and logged citation counts). Additionally, the published articles that did not receive any recommendations to reject from the reviewers received higher citation counts on average (162.8 and 4.72) relative to those with at least one recommendation to reject (115.24 and 4.33). Comparing these numbers to the citation counts of the rejected articles, we can see a rather larger difference: articles being accepted by the high-end journals tended to garner substantially more citations than the ones that were rejected, whether before or after peer review.

That said, there’s a complication present in all this: papers rejected from the most prestigious journals tend to subsequently get published in less-prestigious outlets, which fewer people tend to read. As fewer eyes tend to see papers published in less-cited journals, this might mean that even good articles published in worse journals receive less attention. Indeed, the impact factor of the journal (the average citation count of the recent articles published in it) in which an article was published correlated 0.54 with citation and 0.42 with logged citation counts. To help get around that issue, the authors compared the published to rejected-then-published papers in journals with an impact factor of 8 or greater. When they did so, the authors found, interestingly, that the rejected articles were actually cited more than the accepted ones (212.77 vs 143.22 citations and 4.77 and 4.53 logged citations). While such an analysis might bias the number of “mistaken” rejections upwards (as it doesn’t count the papers that were “correctly” bumped down into lower journals), it’s a worthwhile point to bear in mind. It suggests that, above a certain threshold of quality, the acceptance or rejection by a journal might reflect chance differences more than meaningful ones.

But what about the superstar papers? Of the 15 most cited papers, 12 of them (80%) had been desk rejected. As the authors put it, “This finding suggests that in our case study, articles that would eventually become highly cited were roughly equally likely to be desk-rejected as a random submission“. Of the remaining three papers, two had been rejected after review (one of which had been rejected by two of the top 3 journals in question). While it was generally the case, then, that peer review appears to help weed out the “worst” papers, the process does not seem to be particularly good at recognizing the “best” work. Much like The Beatles Decca audition, then, rockstar papers are not often recognized as such immediately. Towards the end of the paper, the authors make reference to some other notable cases of important papers being rejected (one of which being rejected twice for being trivial and then a third time for being too novel).

“Your blindingly-obvious finding is just too novel”

It is worth bearing in mind that academic journals are looking to do more than just publish papers that will have the highest citation count down the line: sometimes good articles are rejected because they don’t fit the scope of the journal; others are rejected just because the journals just don’t have the space to publish them. When that happens, they thankfully tend to get published elsewhere relatively soon after; though “soon” can be a relative term for academics, it’s often within about half a year.

There are also cases where papers will be rejected because of some personal biases on the part of the reviewers, though, and those are the cases most people agree we want to avoid. It is then that the gatekeepers of scientific thought can do the most damage in hindering new and useful ideas because they find them personally unpalatable. If a particularly good idea ends up published in a particularly bad journal, so much the worse for the scientific community. Unfortunately, most of those biases remain hidden and hard to definitively demonstrate in any given instance, so I don’t know how much there is to do about reducing them. It’s a matter worth thinking about.

References: Siler, K., Lee, K., & Bero, L. (2014). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences (US), DOI10.1073/pnas.1418218112

Why Do People Care About Race?

Posted on by

As I have discussed before, claims about a species’ evolutionary history – while they don’t directly test functional explanations – can be used to inform hypotheses about adaptive function. A good example of this concerns the topic of race, which happens to have been on many people’s minds lately. Along with sex and age, race tends to be encoded by our minds relatively automatically: these are the three primary factors people tend to notice and remember about others immediately. What makes the automatic encoding of race curious is that, prior to the advent of technologies for rapid transportation, our ancestors were unlikely to have consistently traveled far enough in the world to encounter people of other races. If that was the case, then our minds could not possess any adaptations that were selected to attend to it specifically. That doesn’t mean that we don’t attend to race (we clearly do), but rather that the attention that we pay to it is likely the byproduct of cognitive mechanisms designed to do other things. If, through some functional analysis, we were to uncover what those other things were, this could have some important implications for removing, or at least minimizing, all sorts of nasty racial prejudices.

…in turn eliminating the need to murder others for that skin-suit…

This, of course, raises the question what the cognitive mechanisms that end up attending to race have been selected to do; what their function is. One plausible candidate explanation put forth by Kurzban, Tooby, & Cosmides, (2001) is that the mechanisms that are currently attending to race might actually have been designed to attend instead to social coalitions. Though our ancestors might not have traveled far enough to encounter people of different races, they certainly did travel far enough to encounter members of other groups. Our ancestors also had to successfully manage within-group coalitions; questions concerning who happens to be who’s friends and enemies. Knowing the group membership of an individual is a rather important piece of information: it can inform you as to their probability of providing you with benefits or, say, a spear to the chest, among other things. Accordingly, traits that allowed individuals to determine other’s probable group membership, even incidentally, should be attended to, and it just so happens that race gets caught up in that mix in the modern day. That is likely due to shared appearance reflecting probable group memberships; just ask any clique of high school children who dress, talk, and act quite similarly to their close friends.

Unlike sex, however, people’s relevant coalitional membership is substantially more dynamic over time. This means that shared physical appearance will not always be a valid cue for determining who is likely to be siding with who. In such instances, then, we should predict that race-based cues should be disregarded in favor of more predictive ones. In simple terms, then, the hypothesis on the table is that (a) race tends to be used by our minds as a proxy for group membership, so (b) when more valid cues for group membership are present, people should pay much less attention to race.

So how does one go about testing such an idea? Kurzban, Tooby, & Cosmides, (2001) did so by using a memory confusion protocol. In such a design, participants are presented with a number of photos of people, as well as a sentence that the pictured individuals are said to have spoken to each other during a conversation about a sporting dispute they had last year. Following that, participants are given a surprise recall task, during which they are asked to match the sentences to the pictures of the people who said them. The underlying logic is that participants will tend to make a certain pattern of mistakes in their matching: they will confuse individuals with each other more readily to the extent that their mind has placed them in the same group (or, perhaps more accurately, to the extent that their mind has failed to encode differentiating features of the individuals). Framed in terms of race, we might expect that people will mistake a quote attributed to one black person with another, as they had been mentally grouped together, but will be less likely to mistake that quote for one attributed to a white person. Again, the question of interest here is how our minds might be grouping people: is it doing so on the basis of race per se, or on the basis of coalitions?

“Yes; it’s Photoshopped. And yes; you’re racist for asking”

In the first experiment, 8 pictures were presented, split evenly between young white and black males. From the verbal statements that accompanied each picture, they could be classified into one of two coalitions, though participants were not explicitly instructed to attend to that variable. All the men were dressed identically. In this condition, while subjects did appear to pick up on the coalition factor – evidenced by their being somewhat more likely to mistake people who belonged to same coalition with one another – the size of the race effect was twice as large. In other words, when the only cue to group membership was the statement accompanying each picture, people were more likely to mistake one white man for another more often than they were to mistake one member of a coalition for another.

In the second experiment, however, participants were given the same pictures, but now there was an additional visual cue to group membership: half of the men were wearing yellow jerseys while the other half wore gray. In this case, the color of the shirt predicted which coalition each man was in, but participants were again not told to pay attention to that explicitly. In this condition, the previous effect reversed: the size of the race effect was only half that of the effect for coalition membership. It seemed that giving people an alternative visual cue for group membership dramatically cut the race effect. In fact, in a follow-up study reported by the paper (using pictures of different men), the race effect disappeared. When provided with alternate visual cues to coalition membership, people seemed to be largely (though not necessarily entirely) disregarding race. This finding demonstrates that racial categorization is not always automatic and strong as it had previously been thought it to be.

Importantly, when this experiment was run using sex instead of race (i.e., 4 women and 4 men), the above effects did not replicate. Whether the cues to group membership were only verbal or whether they were verbal and visual, people continued to encode sex automatically and do so robustly, as evidenced again by their pattern of mistakes. Though white women and black men are both visually distinct from white men, additional visual cues to coalition membership only had an appreciable effect on latter group, consistent with the notion that the tendency people have to encode race is a byproduct of our coalitional psychology.

“With a little teamwork – black or white – we can all crush our enemies!”

The good news, then, is that people aren’t inherently racist; our evolutionary history wouldn’t allow it, given how far our ancestors likely traveled. We’re certainly interested in coalitions, these coalitions are frequently used to benefit our allies at the expense of non-members, and that part probably isn’t going away anytime soon, but that has a less morally-sinister tone to it for some reason. It is worth noting that, in the reality outside the lab, coalitions may well (and frequently seem to) form among racial or ethnic lines. Thankfully, as I mentioned initially, coalitions are also fluid things, and it (sometimes) only seems to take a small exposure to other visual indicators of membership to change the way people are viewed by others in that respect. Certainly useful information for anyone looking to reduce the impact of race-based categorization.

References: Kurzban, R., Tooby, J., & Cosmides, L. (2001). Can race be erased? Coalitional computation and social categorization. PNAS, 98, 15387-15392.

#HandsUp (Don’t Press The Button)

Posted on by

In general, people tend to think of themselves as not possessing biases or, at the very least, less susceptible to them than the average person. Roughly paraphrasing from Jason Weeden and Robert Kurzban’s latest book, when it comes to debates, people from both sides tend to agree with the premise that one side of the debate is full of reasonable, dispassionate, objective folk and the other side is full of biased, evil, ignorant ones; the only problem is that people seem to disagree as to which side is which. To quote directly from Mercier & Sperber (2011): “[people in debates] are not trying to form an opinion: They already have one. Their goal is argumentative rather than epistemic, and it ends up being pursued at the expense of epistemic soundness” (p.67). This is a long-winded way of saying that people – you and I included – are biased, and we typically end up seeking to support views we already hold. Now, recently, owing to the events that took place in Ferguson, a case has been made that police officers (as well as people in general) are biased against the black population when it comes to criminal justice. This claim is by no means novel; NWA, for instance, voiced in 1988 in their hit song “Fuck tha police”.

 They also have songs about killing people, so there’s that too…

Is the justice system and its representatives, at least in here in the US, biased against the black population? I suspect that most of you reading this already have an answer to that question which, to you, likely sounds pretty obvious. Many people have answered that question in the affirmative, as evidenced by such trending twitter hashtags as #BlackLivesMatter and #CrimingWhileWhite (the former implying that people devalue black lives and the latter implying that people get away with crimes because they’re white, but they wouldn’t if they were black). Though I can’t speak to the existence or extent of such biases – as well as the contexts in which they occur – I did come across some interesting research recently that deals with a related, but narrower question. This research attempts to answer a question that many people feel they already have the answer to: are police officers (or people) quicker to deploy deadly force against black targets, relative to white targets? I suspect many of you anticipate – correctly – that I’m about to tell you that some new research shows people aren’t biased against the black population in that respect. I further suspect that upon hearing that, one of your immediate thoughts will be to figure out why the conclusion must be incorrect.

The first of these papers (James, Vila, & Daratha, 2013) begins by noting that some previous research on the topic (though by no means all) has concluded that a racial bias against blacks exists when it comes to the deployment of deadly force. How did they come to this conclusion? Experimentally, it would seem they used a research method similar to the Implicit Association Task (or IAT): they have participants come into a lab, sit in front of a computer, and ask them to press a “shoot” button when they see armed targets pop up on screen and a “don’t shoot” button when the target isn’t armed. James, Vila, & Daratha (2013) argue that such a task is, well, fairly artificial and, as I have discussed before, artificial tasks can lead to artificial results. Part of that artificiality is that there is no difference between the two responses in such an experiment: both responses just involve pushing one button or another. By contrast, actually shooting someone involves unholstering a weapon and pulling a trigger, while not shooting at least does not involve that last step.So shooting is an action; not shooting is an inaction; pressing buttons, however, are both actions, and simple ones. Further, sitting at a computer and seeing static images pop up on the screen is just a bit less interactive than most police encounters that lead to the use of deadly force. So, whether these results concern people’s biases against blacks translate to anywhere outside the lab is an open question.

Accordingly, what the authors of the current paper did involved what must have been quite the luxurious lab set up. The researchers collected data from around 60 civilians and 40 police and military subjects. During each trial, the subjects were standing in an enclosed shooting range with a large screen that would display a simulations where they might or might not have to shoot. Each subject was provided with a modified Glock pistol (that shot lasers instead of bullets), holsters, and instructions on how to use them. The subjects each went through in between 10-30 simulations that recreated instances where officers had been assaulted or killed; simulations which included naturalistic filming with paid actors (as opposed to the typical static images). The subjects were supposed to shoot the armed targets in the simulation and avoid shooting unarmed ones. As usual, the race of the targets was varied to be white, black, or hispanic, as well as whether or not the targets were armed.

Across three studies, a clear pattern emerged: the participants were actually slower to shoot the armed black targets by in between 0.7 – 1.35 seconds, on average; no difference was found between the white and hispanic targets. This result held for both the civilians and the police. The pattern of mistakes people made was even more interesting: when they shot unarmed targets, they tended to shoot the unarmed black targets less than the unarmed white or hispanic targets; often substantially less. Similarly, subjects were also more likely to fail to shot an armed black target. To the extent that people were making errors or slowing down, they were doing so in favor of black targets, contrary to what many people shouting things right now would predict.

“That result is threatening my worldview; shoot it!”

As these studies appear to use a more realistic context when it comes to shooting – relative to sitting at a computer and pressing buttons – it casts some doubt as whether the previous findings that were uncovered when subjects were sitting at computer screens are able to be generalized to the wider world. Casting further doubt on the validity of the computer-derived results, a second paper by James, Klinger, & Vila (2014) examined the relationship between these subconscious race-base biases and the actual decision to shoot. They did so by reanalyzing some of the data (n = 48) from the previous experiment when participants had been hooked up to EEGs at the time. The EEG equipment was measuring what the authors call “alpha suppression”. According to their explanation (I’m not a neuroscience expert, so I’m only reporting what they do), the alpha waves being measured by the EEG tend to occur when individuals are relaxed, and reductions of alpha waves are associated with the presence of arousing external stimuli; in this case, the perception of threat. The short version of this study, then, seems to be that reductions in alpha waves equate, in some way, to more perception of threat.

The more difficult shooting scenarios resulted in greater alpha suppression than the simpler ones, consistent with a relation to threat level but, regardless of the scenario difficulty, the race effect remained consistent. The EEG results found that, when faced with a black target, subjects evidenced greater alpha suppression relative to when they confronting a white or hispanic target; this result obtained regardless of whether the target ended up being armed or not. To the extent that these alpha waves are measuring threat response on a physiological level, people found the black targets more threatening, but this did not translate into an increased likelihood to shoot them; in fact, it seemed to do the opposite. The authors suggest that this might have something to do with the perception of possible social and legal consequences for harming a member of a historically oppressed racial group.

In other words, people might not be shooting because they’re afraid that people will claim that the shooting was racially motivated (indeed, if the results had turned out the opposite way, I suspect many people would be making that precise claim, so they wouldn’t be wrong). The authors provide some reason to think the social concerns of shooting might be driving the hesitation, one of which involves this passage from an interview of a police chief in 1992:

“Bouza…. added that in most urban centers in the United States, when a police chief is called “at three in the morning and told, ‘Chief, one of our cops just shot a kid,’ the chief’s first questions are: ‘What color is the cop? What color is the kid?’” “And,” the reporter asked, “if the answer is, ‘The cop is white, the kid is black’?” “He gets dressed,”

“I’m not letting a white on white killing ruin this nap”

Just for some perspective, the subjects in this second study had responded to about 830 scenarios in total. Of those, there were 240 that did not require the use of force. Of those 240, participants accidentally shot a total of 47 times; 46 of those 47 unarmed targets were white (even though around a third of the targets were black). If there was some itchy trigger finger concerning black threats, it wasn’t seen in this study. Another article I came across (but have not fact checked so, you know, caveat there) suggests something similar: that biases against blacks in the criminal justice system don’t appear to exist.

Now the findings I have presented here may, for some reason, be faulty. Perhaps better experiments in the future will provide more concrete evidence concerning racial biases, or lack thereof. However, if you first reaction to these findings is to assume that something is wrong with them because you know that police target black suspects disproportionately, then I would urge you to consider that, well, maybe some biases are driving your reaction. That’s not to say that others aren’t biased, mind you, or that you’re necessarily wrong, just that you might be more biased than you like to imagine.

References: James, L., Vila, B. & Daratha, K. (2013) Influence of suspect race and ethnicity on decisions to shoot in high fidelity deadly force judgment and decision-making simulations. Journal of Experimental Criminology, 9, 189–212.

 James, L., Klinger, D., & Vila, B. (2014). Racial and ethnic bias in decisions to shoot seen through a stronger lens: Experimental results from high-fidelity laboratory simulations. Journal of Experimental Criminology, 10, 323-340.