By now, many of you have no doubt heard about the reproducibility project, where 100 psychological findings were subjected to replication attempts. In case you’re not familiar with it, the results of this project were less than a ringing endorsement of research in the field: of the expected 89 replications, only 37 were obtained and the average size of the effects fell dramatically; social psychology research in particular seemed uniquely bad in this regard. This suggests that, in many cases, one would be well served by taking many psychological findings with a couple grains of salt. Naturally, this leads many people to wonder whether there’s anyway they might be more confident that an effect is real, so to speak. One possible means through which your confidence might be bolstered is whether or not the research in question contains conceptual replications. What this refers to are cases where the authors of a manuscript report the results of several different studies purporting to measure the same underlying thing with varying methods; that is, they are studying topic A with methods X, Y, and Z. If all of these turn up positive, you ought to be more confident that an effect is real. Indeed, I have had a paper rejected more than once for only containing a single experiment. Journals often want to see several studies in one paper, and that is likely part of the reason why: a single experiment is surely less reliable than multiple ones.

It doesn’t go anywhere, but at least it does so reliably

According to the unknown moderator account of replication failure, psychological research findings are, in essence, often fickle. Some findings might depend on the time of day that measurements were taken, the country of the sample, some particular detail of the stimulus material, whether the experimenter is a man or a woman; you name it. In other words, it is possible that these published effects are real, but only occur in some rather specific contexts of which we are not adequately aware; that is to say they are moderated by unknown variables. If that’s the case, it is unlikely that some replication efforts will be successful, as it is quite unlikely that all of the unique, unknown, and unappreciated moderators will be replicated as well. This is where conceptual replications come in: if a paper contains two, three, or more different attempts at studying the same topic, we should expect that the effect they turn up is more likely to extend beyond a very limited set of contexts and should replicate more readily.

That’s a flattering hypothesis for explaining these replication failures; there’s just not enough replication going on prepublication, so limited findings are getting published as if they were more generalizable. The less-flattering hypothesis is that many researchers are, for lack of a better word, cheating by employing dishonest research tactics. These tactics can include hypothesizing after data is collected, only collecting participants until the data says what the researchers want and then stopping, splitting samples up into different groups until differences are discovered, and so on. There’s also the notorious issue of journals only publishing positive results rather than negative ones (creating a large incentive to cheat, as punishment for doing so is all but non-existent so long as you aren’t just making up the data). It is for these reasons that requiring the pre-registering of research – explicitly stating what you’re going to look at ahead of time – drops positive findings markedly. If research is failing to replicate because the system is being cheated, more internal replications (those from the same authors) don’t really help that much when it comes to predicting external replications (those conducted by outside parties). Internal replications just provide researchers the ability to report multiple attempts at cheating.

These two hypotheses make different predictions concerning the data from the aforementioned reproducibility project: specifically, research containing internal replications ought to be more likely to successfully replicate if the unknown moderator hypothesis is accurate. It certainly would be a strange state of affairs from a “this finding is true” perspective if multiple conceptual replications were no more likely to prove reproducible than single-study papers. It would be similar to saying that effects which have been replicated are no more likely to subsequently replicate than effects which have not. By contrast, the cheating hypothesis (or, more politely, questionable research practices hypothesis) has no problem at all with the idea that internal replications might prove to be as externally replicable as single-study papers; cheating a finding out three times doesn’t mean it’s more likely to be true than cheating it out once.

It’s not cheating; it’s just a “questionable testing strategy”

This brings me to a new paper by Kunert (2016) who reexamined some of the data from the reproducibility project. Of the 100 original papers, 44 contained internal replications: 20 contained just one replication, 10 were replicated twice, 9 were replicated 3 times, and 5 contained more than three. These were compared against the 56 papers which did not contain internal replications to see which would subsequently replicate better (as measured by achieving statistical significance). As it turned out, papers with internal replications externally replicated about 30% of the time, whereas papers without internal replications externally replicated about 40% of the time. Not only were the internally-replicated papers not substantially better, they were actually slightly worse in that regard. A similar conclusion was reached regarding the average effect size: papers with internal replications were no more likely to subsequently contain a larger effect size, relative to papers without such replications.

It is possible, of course, that papers containing internal replications are different than papers which do not contain such replications. This means it might be possible that internal replications are actually a good thing, but their positive effects are being outweighed by other, negative factors. For example, someone proposing a particularly novel hypothesis might be inclined to include more internal replications in their paper than someone studying an established one; the latter researcher doesn’t need more replications in his paper to get it published because the effect has already been replicated in other work. Towards examining this point, Kunert (2016) made use of the 7 identified reproducibility predictors from the Open Science Collaboration – field of study, effect type, original P-value, original effect size, replication power, surprisingness of original effect, and the challenge of conducting the replication – to assess whether internally-replicated work differed in any notable ways from the non-internally-replicated sample. As it turns out, the two samples were pretty similar overall on all the factors except one: field of study. Internally-replicated effects tended to come from social psychology more frequently (70%) than cognitive psychology (54%). As I mentioned before, social psychology papers did tend to replicate less often. However, the unknown moderator effect was not particularly well supported for either field when examined individually.

In summary, then, papers containing internal replications were no more likely to do well when it came to external replications which, in my mind, suggests that something is going very wrong in the process somewhere. Perhaps researchers are making use of their freedom to analyze and collect data as they see fit in order deliver the conclusions they want to see; perhaps journals are preferentially publishing the findings of people who got lucky, relative to those who got it right. These possibilities, of course, are not mutually exclusive. Now I suppose one could continue to make an argument that goes something like, “papers that contain conceptual replications are more likely to be doing something else different, relative to papers with only a single study,” which could potentially explain the lack of strength provided by internal replications, and whatever that “something” is might not be directly tapped by the variables considered in the current paper. In essence, such an argument would suggest that there are unknown moderators all the way down.

“…and that turtle stands on the shell of an even larger turtle…”

While it’s true enough that such an explanation is not ruled out by the current results, it should not be taken as any kind of default stance on why this research is failing to replicate. The “researchers are cheating” explanation strikes me as a bit more plausible at this stage, given that there aren’t many other obvious explanations for why ostensibly replicated papers are no better at replicating. As Kunert (2016) plainly puts it:

This report suggests that, without widespread changes to psychological science, it will become difficult to distinguish it from informal observations, anecdotes and guess work.

This brings us to the matter of what might be done about the issue. There are procedural ways of attempting to address the problem – such as Kunert’s (2016) recommendation for getting journals to publish papers independent of their results – but my focus has, and continues to be, on the theoretical aspects of publication. Too many papers in psychology get published without any apparent need for the researchers to explain their findings in any meaningful sense; instead, they usually just restate and label their findings, or they posit some biologically-implausible function for what they found. Without the serious and consistent application of evolutionary theory to psychological research, implausible effects will continue to be published and subsequently fail to replicate because there’s otherwise little way to tell whether a finding makes sense. By contrast, I find it plausible that unlikely effects can be more plainly spotted – by reviewers, readers, and replicators – if they are all couched within the same theoretical framework; even better, the problems in design can be more easily identified and rectified by considering the underlying functional logic, leading to productive future research.  

References: Kunert, R. (2016). Internal conceptual replications do not increase independent replication success. Psychological Bulletin Review, DOI 10.3758/s13423-016-1030-9

Let’s talk research ethics for a moment.

Would you rather have someone actually take $20 from your payment for taking part in a research project, or would you rather be told – incorrectly – that someone had taken $20, only to later (almost immediately, in fact) find out that your money is safely intact and that the other person who supposedly took it doesn’t actually exist? I have no data on that question, but I suspect most people would prefer the second option; after all, not losing money tends to be preferable to losing money, and the lie is relatively benign. To use a pop culture example, Jimmy Kimmel has aired a segment where parents lie to their children about having eaten all their Halloween candy. The children are naturally upset for a moment and their reactions are captured so people can laugh at them, only to later have their candy returned and the lie exposed (I would hope). Would it be more ethical, then, for parents to actually eat their children’s candy so as to avoid lying to their children? Would children prefer that outcome?

“I wasn’t actually going to eat your candy, but I wanted to be ethical”

I happen to think that answer is, “no; it’s better to lie about eating the candy than to actually do it” if you are primarily looking out for the children’s welfare (there is obviously the argument to be made that it’s neither OK to eat the candy or to lie about it, but that’s a separate discussion). That sounds simple enough, but according to some arguments I have heard, it is unethical to design research that, basically, mimics the lying outcome. The costs being suffered by participants need to be real in order for research on suffering costs to be ethically acceptable. Well, sort of; more precisely, what I’ve been told is that it’s OK to lie to my subjects (deceive them) about little matters, but only in the context of using participants drawn from undergraduate research pools. By contrast, it’s wrong for me to deceive participants I’ve recruited from online crowd-sourcing sites, like Mturk. Why is that the case? Because, as the logic continues, many researchers rely on MTurk for their participants, and my deception is bad for those researchers because it means participants may not take future research seriously. If I lied to them, perhaps other researchers would too, and I have poisoned the well, so to speak. In comparison, lying to undergraduates is acceptable because, once I’m done with them, they probably won’t be taking part in many future experiments, so their trust in future research is less relevant (at least they won’t take part in many research projects once they get out of the introductory courses that require them to do so. Forcing undergraduates to take part in research for the sake of their grade is, of course, perfectly ethical).

This scenario, it seems, creates a rather interesting ethical tension. What I think is happening here is that a conflict has been created between looking out for the welfare of research participants (in common research pools; not undergraduates) and looking out for the welfare of researchers. On the one hand, it’s probably better for participants’ welfare to briefly think they lost money, rather than to let them actually lose money; at least I’m fairly confident that is the option subjects would select if given the choice. On the other hand, it’s better for researchers if those participants actually lose money, rather than briefly hold the false believe that they did, so participants continue to take their other projects seriously. An ethical dilemma indeed, balancing the interests of the participants against those of the researchers.

I am sympathetic to the concerns here; don’t get me wrong. I find it plausible to suggest that if, say, 80% of researchers outright deceived their participants about something important, people taking this kind of research over and over again would likely come to assume some parts of it were unlikely to be true. Would this affect the answers participants provide to these surveys in any consistent manner? Possibly, but I can’t say with any confidence if or how it would. There also seems to be workarounds for this poisoning-the-well problem; perhaps honest researchers could write in big, bold letters, “the following research does not contain the use of deception” and research that did use deception would be prohibited from attaching that bit by the various institutional review boards that need to approve these projects. Barring the use of deception across the board would, of course, create its own set of problems too. For instance, many participants taking part in research are likely curious as to what the goals of the project are. If researchers were required to be honest and transparent about their purposes upfront so as to allow their participants to make informed decisions regarding their desire to participate (e.g., “I am studying X…”), this can lead to all sorts of interesting results being due to demand characteristics - where participants behave in unusual manners as a result of their knowledge about the purpose of the experiment – rather than the natural responses of the subjects to the experimental materials. One could argue (and many have) that not telling participants about the real purpose of the study is fine, since it’s not a lie as much as an omission. Other consequences of barring explicitly deception exist as well, though, including the lack of control over experimental stimuli during interactions between participants and the inability to feasibly even test some hypotheses (such as whether people prefer the tastes of identical foods, contingent on whether they’re labeled in non-identical ways).

Something tells me this one might be a knock off

Now this debate is all well and good to have in the abstract sense, but it’s important to bring some evidence to the matter if you want to move the discussion forward. After all, it’s not terribly difficult for people to come up with plausible-sounding, but ultimately incorrect, lines of reasoning as for why some research practice is possibly (un)ethical. For example, some review boards have raised concerns about psychologists asking people to take surveys on “sensitive topics”, under the fear that answering questions about things like sexual histories might send students into an abyss of anxiety. As it turns out, such concerns were ultimately empirically unfounded, but that does not always prevent them from holding up otherwise interesting or valuable research. So let’s take a quick break from thinking about how deception might be harmful in the abstract to see what effects it has (or doesn’t have) empirically.

Drawn by the debate between economists (who tend to think deception is bad) and social scientists (who tend to think it’s fine), Barrera & Simpson (2012) conducted two experiments to examine how deceiving participants affected their future behavior. The first of these studies tested the direct effects of deception: did deceiving a participant make them behave differently in a subsequent experiment? In this study, participants were recruited as part of a two-phase experiment from introductory undergraduate courses (so as to minimize their previous exposure to research deception, the story goes; it just so happens they’re likely also the easiest sample to get). In the first phase of this experiment, 150 participants played a prisoner’s dilemma game which involved cooperating with or defecting on another player; a decision which would affect both player’s payments. Once the decisions had been made, half the participants were told (correctly) that they had been interacting with another real person in the other room; the other half were told they had been deceived, and that no other player was actually present. Everyone was paid and sent home.

Two to three weeks later, 140 of these participants returned for phase two. Here, they played 4 rounds of similar economic games: two rounds of dictator-games and two rounds of trust-games. In the dictator games, subjects could divide $20 between themselves and their partner; in the trust games, subjects could send some amount of $10 to the other player, this amount would be multiplied by three, and that player could then keep it all or send some of it back. The question of interest, then, is whether the previously-deceived subjects would behave any differently, contingent on their doubts as to whether they were being deceived again. The thinking here is that if you don’t believe you’re interacting with another real person, then you might as well be more selfish than you otherwise would. The results showed that while the previously-deceived participants were more likely to believe that social science researchers used deception somewhat more regularly, relative to the non-deceived participants their behavior was actually no different. Not only were the amounts of money sent to others no different (participants gave $5.75 on average in the dictator condition and trusted $3.29 when they were not previously deceived, and gave $5.52 and trusted $3.92 when they had been), but the behavior was no more erratic either. The deceived participants behaved just like the non-deceived ones.

In the second study the indirect effects of deception were examined. One-hundred-six participants first completed the same dictator and trust games as above. They were then either assigned to read about an experiment that did or did not make use of deception; a deception which included the simulation of non-existent participants. They then played another round of dictator and trust games immediately afterwards to see if their behavior would differ, contingent on knowing about how researchers might be deceive them. As in the first study, no behavioral differences emerged. Neither directly deceiving participants about the presence of others in the experiment or providing them with information that deception does take place in such research seemed to have any noticeable effects on subsequent behavior.

“Fool me once, shame on me; Fool me twice? Sure, go ahead”

Now it is possible that the lack of any effect in the present research had to do with the fact that participants were only deceived once. It is certainly possible that repeated exposures to deception, if frequent enough, will begin to have an effect and that effect will be a lasting one and it will not just be limited to the researcher employing the deception. In essence, it is possible that some spillover between experimenters over time might occur. However, this is something that needs to be demonstrated; not just assumed. Ironically, as Barrera & Simpson (2012) note, demonstrating such a spillover effect can be difficult in some instances, as designing non-deceptive control conditions to test against the deceptive ones is not always a straightforward task. In other words, as I mentioned before, some research is quite difficult – if not impossible – to conduct without being able to use deception. Accordingly, some control conditions might require that you deceive participants about deceiving them, which is awfully meta. Barrera & Simpson (2012) also mention some research findings that report even when no deception is used, participants who repeatedly take part in these kinds of economic experiments tend to get less cooperative over time. If that finding holds true, then the effects of repeated deception need to be filtered out from the effects of repeated participation in general. In any case, there does not appear to any good evidence that minor deceptions are doing harm to participants or other researchers. They might still be doing harm, but I’d like to see it demonstrated before I accept that they do. 

References: Barrera, D. & Simpson, B. (2012). Much ado about deception: Consequences of deceiving research participants in the social sciences. Sociological Methods & Research, 41, 383-413.

Diversity is one of the big buzzwords of the recent decades. Institutions, both public and private, often take great pains to emphasize their inclusive stances and colorful cast of a staff. I have long found the displays of diversity to be rather queer in one major respect, however: they almost always focus on diversity in the realms of race and gender. The underlying message behind such displays would seem to suggest that men and women, or members of different ethnic groups, are, in some relevant psychological respects, different from one another. What’s strange about that idea is that, as many of the same people might also like to point out, there’s less diversity between those groups than within them, while others are entirely uncomfortable with the claim of sex or racial differences from the start. The ambivalent feelings many people have surrounding such a message were captured well by Principle Skinner on The Simpsons:

“It’s the differences…of which…there are none…that make the sameness… exceptional“

Regardless of how one feels about such a premise, the fact remains that diversity in race or gender per se is not what people are seeking to maximize in many cases; they’re trying to increase diversity of thought (or, as Maddox put it many years ago: “people who look different must think different because of it; otherwise, why the hell embrace anything? Why not just assume that diversity comes from within, regardless of their skin color, sex, age or religion?“)

Renting that wheel chair was a nice touch, but it’s time to get up and return it before we lose the deposit

If diversity in perspective is what most people are after when they talk about seeking diversity, it seems like it would be a reasonable step to assess people’s perspectives directly, rather than trying to use proxies for it, like race and gender (or clothing, or hair styles, or musical tastes, or…). If, for instance, one was hiring a number of people for a job involving problem solving, it’s quite possible for the person doing the hiring to select a group of men and women from different races who all end up thinking about things in pretty much the same way: not only would the hires likely have the same kinds of educational background, but they’d probably also have comparable interests since they applied for the same job. On top of that initial similarity the person doing the hiring might be partial towards those who hold agreeable points of view. After all, why would you hire someone who holds a perspective you don’t agree with? It sounds as if that decision would make work that much more unpleasant during the day-to-day operations of the company, even if it was irrelevant to the work they do.

Speaking of areas in which diversity of thought seem to be lacking in certain respects, an interesting new paper from Duarte et al (2015) puts forth the proposition that social psychology – as a field – isn’t all that politically diverse, and that’s probably something of a problem for research quality. For example, if social psychologists can be said to be a rather politically homogeneous bunch, this could result in particular (and important) questions not being asked as a result of how that answer might pan out for the images of liberals and their political rivals. After all, if the conclusions of psychology research, by some happy coincidence, tend to demonstrate that liberals (and, by extension, the liberal researchers conducting it) happen to have a firm grasp on reality, whereas their more conservative counterparts are hopelessly biased and delusional, all the better for the liberal group’s public image; all the worse for the truth value of psychological research, however, if those results are obtained by only asking about scenarios in which conservatives, but not liberals, are likely to look biased. If some liberal assumptions about what is right or good are shaping their research to point in certain directions, we’re going to end up making a number of unwarranted interpretative conclusions.

The problems could mount further if the research purporting to deliver conclusions counter to certain liberal interests is reviewed with disproportionate amounts of scrutiny, whereas research supporting those interests is given a pass when their methods are equivalent or worse. Indeed, Duarte et al (2015) discuss some good reasons to think this might be the state of affairs in psychology, not least of which is that quite a number of social psychologists will explicitly admit they would discriminate against those who do not share their beliefs. When surveyed about their self-assessed probability of voting either for or against a known conservative job applicant (when both alternatives are equally qualified for the job), about 82% of social psychologists indicated they would be at least a little more likely to vote against the conservative hire, with about 43% indicating a fairly high degree of certainty they would (above the midpoint of the scale). These kinds of attitudes might well dissuade more conservatives from wanting to enter the field, especially given that the liberals likely to discriminate against them outnumber the conservatives by about 10-to-1.

“Don’t worry, buddy; you can take ‘em”

Not to put too fine of a point on it, but if these ratios were discovered elsewhere – say, a 10:1 ratio of men to women in a field, and about half of the men explicitly say they would vote against hiring women – I imagine that many social psychologists would tripping over themselves to try and inject some justice and moral outrage into the mix. Compared with some other explicit racist tendencies (4% of respondents wouldn’t vote for a black presidential candidate), or sexist ones (5% wouldn’t vote for a woman), there’s a bit of a gulf in discrimination. While the way the question is asked is not quite the same, social psychologists might be about as likely to want to vote for the conservative job candidate as Americans are to vote for a Muslim or an atheist if we assumed equivalence (which is is to say “not very”).

It is at least promising, then, to see that the reactions to this paper were fairly universal in at least recognizing that there might be something of a political diversity problem in psychology, both in terms of its existence and possible consequences. There was more disagreement with respect to the cause of this diversity problem and whether including more conservative minds would increase research quality, but that’s to be expected. I – like the authors – am happy enough that even social psychologists, by in large, seem to accept that social psychology is not all that politically diverse and that such a state of affairs is likely – or at least potentially – harmful to research in some respects (yet another example where stereotypes seem to track reality well).

That said, there is another point to which I want to draw attention. As I mentioned initially, seeking diversity for diversity’s sake is a pointless endeavor, and one that is certainly not guaranteed to improve the quality of work produced. This is the case regardless of the criteria on which candidates are selected, be they physical, political, or something else.  For example, psychology departments could strive to hire people from a variety of different cultural or ethnic groups, but unless those new hires are better at doing psychology, this diversity won’t improve their products. Similarly, psychology departments could strive to hire people with degrees in other fields, like computer science, chemistry, and fine arts; that would likely increase the diversity of thought in psychology, but since there are many more ways of doing poor psychology than there are of doing good psychology, this diversity in backgrounds wouldn’t necessarily be desirable.

Say “Hello” to your new collaborators

Put bluntly, I wouldn’t want people to begin hiring those from non-liberal groups in greater numbers and believe this will, de facto, improve the quality of their research. More specifically, while greater political diversity might, to some extent, reduce the number of bad research projects by diluting or checking existing liberal biases, I don’t know that it would increase in the number of good papers substantially; the relative numbers might change, but I’m more concerned with the absolutes, as a field which fails to produce quality research in sufficient quantities is not demonstrating much value (just like how the guy without a particular failing doesn’t necessarily offer much as a dating prospect). In my humble (and no doubt biased, but not necessarily incorrect) view, there is an important dimension of thought along which I do not wish psychologists to differ, and that is in their application of evolutionary theory as a guiding foundation for their work. Evolutionary theory not only allows one to find previously unappreciated aspects of psychological functioning by considerations of adaptive value, but also allows for building on previous research in a meaningful way and for the effective rooting out of problematic underlying assumptions. In that sense, even failed research projects can contribute in a more meaningful way when framed in an evolutionary perspective, relative to failed projects lacking one.

Evolutionary theory is by no means a cure-all for the bias problem; people will still sometimes get caught up trying to rationalize behaviors or preferences they morally approve of – like homosexuality – as adaptive, for example. In spite of that, I do not particularly hope to see a diversity of perspectives in psychology regarding the theoretical language we all ought to speak by this point. There are many more ways to think about psychology unproductively than there are of doing it well, and more diversity in those respects will make for a much weaker science.

References: Duarte , J., Crawford, J., Stern, C., Haidt, J., Jussim, L., & Tetlock, P. (2015). Political diversity will improve social psychological science. Behavioral & Brain Sciences, 38, 1-58.

There are moments from my education that have stuck with me over time. One such moment involved a professor teaching his class about what might be considered a “classic” paper in social psychology. I happened to have been aware of this particular paper for two reasons: first, it was a consistent feature in many of my previous psychology classes and, second, because the news had broke recently that when people tried to replicate the effect they had failed to find it. Now a failure to replicate does not necessarily mean that the findings of the original study were a fluke or the result of experimental demand characteristics (I happen to think they are), but that’s not even why this moment in my education stood out to me. What made this moment stand out is that when I emailed the professor after class to let him know the finding had recently failed to replicate, his response was that he was aware of the failure. This seemed somewhat peculiar to me; if he knew the study had failed to replicate, why didn’t he at least mention that to his students? It seems like rather important information for the students to have and, frankly, a responsibility of the person teaching the material, since ignorance was no excuse in this case.

“It was true when I was an undergrad, and that’s how it will remain in my class”

Stories of failures to replicate have been making the rounds again lately, thanks to a massive effort on the part of hundreds of researchers to try and replicate 100 published effects in three psychology journals. These researchers worked with the original authors, used the original materials, were open about their methods, pre-registered their analyses, and archived all their data. Of these 100 published papers, 97 of them reported their effect as being statistically significant, with the other 3 being right on the borderline of significance and interpreted as being a positive effect. Now there is debate over the value of using these kinds of statistical tests in the first place, but, when the researchers tried to replicate these 100 effects using the statistically significant criterion, only 37 even managed to cross the barrier (given that 89 were expected to replicate if the effects were real, 37 is falling quite short of that goal).

There are other ways to assess these replications, though. One method is to examine the differences in effect size. The 100 original papers reported an average effect size of about 0.4; the attempted replications saw this average drop to about 0.2. A full 82% of the original papers showed a stronger effect size than the attempted replications, While there was a positive correlation (about r = 0.5) between the two – the stronger the original effect, the stronger the replication effect tended to be – this still represents an important decrease in the estimated size of these effects, in addition to their statistical existence. Another method of measuring replication success – unreliable as it might be – is to get the researcher’s subjective opinions about whether the results seemed to replicate. On that front, the researchers felt about 39 of the original 100 findings replicated; quite in line with the above statistical data. Finally, perhaps worth noting, social psychology research tended replicate less often than cognitive research (25% and 50%, respectively), and interaction effects replicated less often than simple effects (22% and 47%, respectively).

The scope of the problem may be a bit larger than that, however. In this case, the 100 papers upon which replication efforts were undertaken were drawn from three of the top journals in psychology. Assuming a positive correlation exists between journal quality (as measured by impact factor) and the quality of research they publish, the failures to replicate here should, in fact, be an underestimate of the actual replication issue across the whole field. If over 60% of papers failing to replicate is putting the problem a bit mildly, there’s likely quite a bit to be concerned about when it comes to psychology research. Noting the problem is only one step in the process towards correction, though; if we want to do something about it, we’re going to need to know why it happens.

So come join in my armchair for some speculation

There are some problems people already suspect as being important culprits. First, there are biases in the publication process itself. One such problem is that journals seem to overwhelmingly prefer to report positive findings; very few people want to read about a bad experiment which didn’t work out well. A related problem, however, is that many journals like to publish surprising, or counter-intuitive findings. Again, this can be attributed to the idea that people don’t want to read about things they already believe are true: most people perceive the sky as blue and research confirming this intuition won’t make many waves. However, I would also reckon that counter-intuitive findings are surprising to people precisely because they are also more likely to be inaccurate descriptions of reality. If that’s the case, than a preference on the part of journal editors for publishing positive, counter-intuitive findings might set them up to publish a lot of statistical flukes.

There’s also the problem I’ve written about before, concerning what are known as “research degrees of freedom“; more colloquially, we might consider this a form of data manipulation. In cases like these, researchers are looking for positive effects, so they test 20 people in each group and peak at the data. If they find an effect, they stop and publish it; if they don’t, they add a few more people and peak again, continuing until they find what they want or run out of resources. They might also split the data up into various groups and permutations until they find a set of data that “works”, so to speak (break it down by male/female, or high/medium/low, etc). While they are not directly faking the data (though some researchers do that as well), they are being rather selective about how they analyze it. Such methods inflate the possibility of finding of effect through statistical brute force, even if the effect doesn’t actually exist.

This problem is not unique to psychology, either. A recent paper by Kaplan & Irvin (2015) examined research from 1970-2012 that was looking at the effectiveness of various drugs and dietary supplements for preventing or treating cardiovascular disease. There were 55 trials that met the author’s inclusion criteria. What’s important to note about these trials is that, prior to the year 2000, none of the papers were pre-registered with respect to what variables they were interested in assessing; after 2000, every such study was pre-registered. Registering this research is important, as it doesn’t allow the researchers to then conduct a selective set of analyses on their data. Sure enough, prior to 2000, 57% of trials reported statistically-significant effects; after 2000, that number dropped to 8%. Indeed, about half the papers published after 2000 did report some statistically significant effects, but only for variables other than the primary outcomes they registered. While this finding is not necessarily a failure to replicate per se, it certainly does make one wonder about the reliability of those non-registered findings.

And some of those trials were studying death as an outcome, so that’s not good…

There is one last problem I would like to mention; one I’ve been beating the drum for for the past several years. Assuming that pre-registering research in psychology would help weed out false positives (it likely would), we would still be faced with the problem that most psychology research would not find anything of value, if the above data are any indication. In the most polite way possible, this would lead me to ask a question along the lines of, “why are so many psychology researchers bad at generating good hypotheses?” A pre-registered bad idea does not suddenly make it a good one, even if it makes data analysis a little less problematic. This leads me to my suggestion for improving research in psychology: the requirement of actual theory for guiding research. In psychology, most theories are not theories, but rather restatements of a finding. However, when psychologists begin to take an evolutionary approach to their work, the quality of research (in my obviously-biased mind) tends to improve dramatically. Even if the theory is wrong, making it explicit allows problems to be more easily discussed, discovered, and corrected (provided, of course, that one understands how to evaluate and test such theories, which many people unfortunately do not). Without guiding/foundational theories, the only thing you’re left with when it comes to generating hypotheses are the existing data and your intuitions which, again, don’t seem to be good guides for conducting quality research.

References: Kaplan, R. & Irvin, V. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One, doi:10.1371/journal.pone.013238