In one of the most interesting short reports I read recently, some research was conducted in Australia examining what the effect of blind reviews would be on hiring. The premise of the research, far as I can surmise, was that a fear existed of conscious or unconscious bias against women and minority groups when it came to getting hired. This bias would naturally make it harder for those groups to find employment, ultimately yielding a less diverse workforce. In the interests of avoiding that bias, the research team compared what happened when candidates were assessed on either standard resumes or de-identified ones. The latter resumes were identical to the former, except they had group-relevant information (like gender and race) removed. If reviewers don’t have that information of race or gender available, then they couldn’t possibly assess the candidates on the basis of them, whether consciously or unconsciously. That seems straightforward enough. The aim was to compare the results from the blind assessments to those of the standard resumes. As it turned out, there were indeed hints of bias; relatively small in size sometimes, but present nonetheless. However, the bias did not go in the direction that had been feared.

Shocking that the headline wasn’t “Blind review processes are biased”

Specifically, when the participants assessing the resumes had information about gender, they were about 3% more likely to select women, and 3% less likely to select men. Further, minorities were more likely to be selected as well when the information was available (about 6% for males and 9% for females). While there’s more to the picture than that, the primary result seemed to be that, when given the option, these reviewers discriminated in favor of women and minority groups simply because of their group membership. If these results had run in the opposite direction (against women and minorities) there would have no doubt been calls for increasing blind reviews. However, because blind reviews seemed to disfavor women and minorities, the authors had a different suggestion:

Overall, the results indicate the need for caution when moving towards ’blind’ recruitment processes in the Australian Public Service, as de-identification may frustrate efforts aimed at promoting diversity

It’s hard to interpret that statement as anything other than ”we should hire more women and minorities, regardless of qualifications.” Even if sex and race ought to be irrelevant to the demands of the job and candidates should be assessed on their merit, people should also apparently be cautious when removing those irrelevant pieces from the application process. The authors seemed to favor discrimination based on sex or race so long as it benefited the right groups. Such discriminatory practices have led to negative reactions on the part of others, as one might expect.

This brings me another question: why should we value diversity when it comes to hiring decisions? To be clear, the diversity being sought is often strictly demographic in nature (many organizations tout diversity in race, for instance, but not in perspective. I don’t recall the draw of many positions being that you will meet a variety of people who hold fundamental disagreements with your view on the world). It’s also usually the kind of diversity that benefits women and minorities (I’ve never come across calls to get more white males into certain fields dominated by women or other races. Perhaps they exist; I just haven’t seen them). But are there real economic benefits to increasing diversity per se? Could it be the case that more diverse organizations just do better? On the face of it, I would assume the answer is “no” if the diversity in question is simply demographic in nature. What matters when it comes to job performance is not the color of one’s skin or what sex chromosomes they possess, but rather their skills and competencies they bring with them. While some of those skills and competencies might be very roughly approximated by race and gender if you have no additional information about your applicants, we thankfully don’t need to rely on those indirect measures. Rather than asking about gender or race, one could just ask directly about skill sets and interests. When you can do that, the additional value of knowing one’s group membership is likely close to nil. Why bother using a predictor of a variable when you can just use the variable itself?

Do you really love roundabouts that much?

Nevertheless, it has apparently been reported before that demographic diversity predicts the relative success of companies (Herring, 2009). A business case was made for diversity, such that diverse companies were found to generally do better than less diverse ones across a number of different metrics. Not that those in favor of increasing diversity really seemed to need a financial justification, but having one certainly wouldn’t hurt their case. As this paper was apparently popular within the literature (for what I assume is that reason), a replication was attempted (Stojmenovska et al, 2017), beginning in a graduate course as an assignment to help students “learn from the best.” Since it seems “psychology research” and “replications” mix about as well as oil and water as of late, the results turned out a bit worse than hoped. The student wasn’t even trying to look for problems; they just stumbled upon them.  

In this instance, the replication attempt failed to find the published result, instead catching two primary mistakes made in the original paper (as opposed to anything malicious): there were a number of coding mistakes within the data, and the sample data itself was skewed. Without going too deeply into why this is a problem, it should suffice to say that coding mistakes are bad for all the obvious reasons. Fixing the coding mistakes by deleting missing data resulted in a substantial reduction in sample size (25-50% smaller). As for the issue of skew, having a skewed sample can result in an underestimation of the relationship between predictors and outcomes. In brief, there were confounding relationships between predictor variables and the outcomes that were not adequately controlled for in the original paper. To correct for the skew issue, a log transformation on the data was carried out, resulting in a dramatic increase in the relationship between particular variables.

In order to provide a concrete sense for that increase, in the original report the correlation between company size and racial diversity was .14; after the log transformation was carried out, that correlation increased to .41. This means that larger companies tended to be more racially diverse than smaller ones, but that relationship was not fully accounted for in the original paper examining how diversity impacted success. The same issue held for gender diversity and establishment size.

Once these two issues – coding errors and skewed data – were addressed, the new results showed that gender and racial diversity were effectively unrelated to company performance. The only remaining relationship was a small one between gender diversity and the logged number of customers. While seven of the original eight hypotheses were supported in the first paper, the replication attempt correcting these errors only found one of the eight to be statistically supported. As most of the effects no longer existed, and the one that did exist was small in size, the business justification for increasing racial and gender diversity failed to receive any real support.

Very colorful, but they ultimately all taste the same

As I initially mentioned, I don’t see a very good reason to expect that a more demographically diverse group of employees should yield better outcomes. They don’t yield worse outcomes either. However, the study from Australia suggests that the benefits of diversity (or the lack thereof) are basically besides the point in many instances. That is, not only would I imagine this failure to replicate won’t have a substantial impact on many people’s views on whether or not diversity should be increased, but I don’t think it would even if diversity was found to be a bad thing, financially speaking. This is because I don’t suspect many views of whether increasing diversity should be done are based on the foundation that it’s good for people economically in the first place. Increasing diversity isn’t viewed as a tricky empirical matter as much as it seems to be a moral one; one in which certain groups of people are viewed as owing or deserving various things.

This is only looking at the outcomes of adding diversity, of course. The causes of such diverse levels of diversity across different walks of life is another beast entirely.

References: Stojmenovska, D., Bol, T., & Leopolda, T. (2017). Does diversity pay? Replication of Herring (2009). American Sociological Review, 82, 857-867. 

Herring, C. (2009). Does diversity pay? Race, gender, and the business case for diversity. American Sociological Review, 74, 208–224.

There’s an idea floating around the world of psychology referred to as Social Comparison Theory. The basic idea is that people want to know how well they measure up to others in the world and so will compare themselves to others. While there’s obviously more to it than that (including some silly suggestions along the lines of people comparing themselves to others to feel better, rather than to do something adaptive with that information), the principle has been harnessed by researchers examining inequality. Specifically, it has been proposed that inequality itself makes people sad. According to the status-anxiety hypothesis, when it comes to things like money and social status, people make a lot of upwards comparisons between themselves and those doing better. Seeing that other people in the world are doing better than them, they become upset, and this is supposed to be why inequality is bad. I think that’s the idea, anyway. Feel free to add in any additional, more-refined versions of the hypothesis if you’re sitting on them.

“People are richer than me,” now warrants a Xanax prescription

As it turns out, that idea seems to be a little less than true. Before getting to that, though, I wanted to make a few general points (or warnings) about the research I’ve encountered on inequality aversion; the idea that people dislike inequality itself and seek to reduce it when possible (especially the kind that leaves them with less than others). The important point to make about research on inequality is that, if you are looking to get a solid measure of the effects of inequality itself, you need to get everything that is not inequality out of your measures. That’s a basic point for any research, really. 

For instance, when examining research on inequality in the past, I kept noticing that the papers on the topic almost always contained confounding details which impeded the ability of the authors to make the interpretations of the data they were interested in making. Several papers looked at the effects of inequality on punishment in taking games. The basic set up here is that you and I would start with some amount of money. I then take some of that money from you for myself. Because I have taken from you, we either end up with you being better off, me being better off, or both of us being equal. After I take from you, you would be given the option to punish me for my behavior and, as it turns out, people preferentially punish when the taker ends up with more money than them. So if I took money from you, you’d be more likely to punish me if I ended up better off, relative to cases where we were equal or you were still better off. (This happens in research settings with experimental demand characteristics, anyway. In a more naturalistic setting when someone mugs you, I can’t imagine many people’s first thoughts are, “He probably needs the money more than me, so this is acceptable.”)

While such research can tell us about the effects of inequality to some extent, it cannot tell us about the effects of inequality that are distinct from taking. To put that in concrete example, my subsequent research (Marczyk, 2017) used that same taking game to replicate the results while adding two other inequality-generating conditions: one in which I could increase my payment with no impact on you, and another where I could decrease your payment at no benefit to myself. In those two conditions, I found that inequality didn’t appear to have any appreciable impact on subsequent punishment: if I wasn’t harming you, then you wouldn’t punish me even if I generated inequality; if I was harming you, you would punish me even if I was worse off. This new piece of information tells us something very important: namely, that people do not consistently want to achieve equality. When we have been harmed, we usually want to punish, even if punishing generates more inequality than originally existed. (That said, there are still demand characteristics in my work worth addressing. Specifically, I’d bet any effects of inequality would be reduced even further when the money the participants get is earned, rather than randomly distributed by an experimenter)

In terms of the research I want to talk about today, this is relevant because this new – and incredibly large – analysis sought to examine the effects of income inequality on happiness as distinct from the overall economic development of a country (Kelley & Evans, 2017). Apparently lots of previous work had been looking at the relationship between inequality within nations and their happiness without controlling for other important variables. The research question of this new work was, effectively, all else being equal, does inequality itself tend to make people unhappy? The simple example of this question they put forth was to imagine twins: John and James. John lives in a country with relatively low income-inequality and makes $20,000 a year. James lives in a country with relatively high income-inequality and makes $20,000 a year. Will John or James be happier? They sought to examine, on the national level, this connection between inequality and life satisfaction/happiness. 

“At least we’re all poor together”

In order to get that all else to be equal, there are a number of things you need to control for that might be expected to affect life satisfaction. The first of these is GDP per capita; how much a nation tends to produce per person. This is important because it might mean a lot less for your happiness that everyone is equal if that equality means everyone lives in extreme poverty. If that happens to the be the case, then increasing industrialization of a nation can actually increase opportunities for economic advancement while also increasing inequality (as the rewards of such a process aren’t shared equally among the population, at least initially. After a time, a greater percentage of the population will begin to share in those rewards and the relationship between inequality and economic development decreases).  

The other factors you need to control for are individual ones. Just because a society might be affluent, that does not mean that the person answering your survey happens to be. This means controlling for personal income as well, as making more money tends to make for happier people. The authors also controlled for known correlates of happiness including sex, age, marriage status, education, and religious attendance. It’s only once all these factors have been controlled for that you can begin to consider the effect of national inequality (as measured by the Gini coefficient) on life satisfaction ratings. That’s not to say these are all the relevant controls, but they’re a pretty good start. 

Enacting these controls is exactly what the researchers did, pooling data from 169 surveys in 68 societies, representing over 200,000 individuals. If there’s a connection between inequality and life satisfaction to be found, it should be evident here. Countries were categorized as a member of either developing nations (those below 30% of the US per-captial GDP) or advanced ones (those above the 30% mark), and the same analyses was run on each. The general findings of the research are easy to summarize: in developing nations, inequality was predictive of an increase in societal happiness (about 8 points on a 1-100 scale); among the advanced nations, there was no relationship between inequality and happiness. This largely appeared to be the case because, as previously outlined, the onset of development in poorer countries generated initial periods of greater inequality. As development advances, however, this relationship disappears.

A separate analysis was also run on families in the bottom 10% of a nation in terms of income, compared with the families in the top 10% since much of the focus on inequality has discussed the divide between the poor and the rich. As expected, rich people tended to be happier than poor ones, but the presence of inequality was, as before, a boon for happiness and life satisfaction in both groups. It was not that inequality made the poor feel bad while the rich felt good. Whatever the reason for this, it does not seem like poor people were looking up at rich people and feeling like their life was terrible because others had more.

“Some day, all this could be yours…”

All this is not to say that inequality itself is going to make people happy as much as the things that inequality represents can. Inequality can signal the onset of industrialization and development, or it can signal there is hope of improving one’s lot in life through hard work. These are positives for life satisfaction. Inequality might also represent that the warlord in the next town over is very good at stealing resources. This would be bad. However, whatever the reason for these correlations, it does not seem to be the case that inequality per se is what makes people unhappy with life (though living in nations with high GDP and earning good salaries seem to put a smile on some faces).

I like this interpretation of the data, unsurprisingly, because it happens to fit well with my own. In my experiments, people didn’t seem to be punishing inequality itself; they were punishing particular types of behaviors – like the stealing or destruction of resources – that just so happened to generate inequality at times. In other words, people are responding primarily to the means through which inequality arises, rather than the inequality itself. This appears to be the case in the present paper as well. Most telling of this interpretation, I feel, is a point mentioned within the paper without much discussion (as its the topic of a separate one): the national data was collected from non-communist nations. Things are a little different in the communist countries. For those cohorts who lived their formative years in communist nations, inequality appears to have a negative relationship with happiness, though that dissipates in new, post-communist generations. From that finding, it seems plausible to speculate that communists might have different ideas about the means through which inequality arises (mostly negative) which they push rather aggressively, relative to non-communists. That said, those attitudes do not seem to persist without consistent training.

Reference: Kelley, J. & Evans, M. (2017). Societal inequality and individual subjective well-being: Results from 68 societies and over 200,000 individuals, 1981-2008. Social Science Research, 62, 1-23.

Marczyk, J. (2017). Human punishment is not primarily motivated by inequality. PLOS One, https://doi.org/10.1371/journal.pone.0171298

This semester I happen to be teaching a course on human learning and memory. Part of the territory that comes with designing and teaching any class is educating yourself on the subject: brushing up on what you do know and learning about what you do not. For the purposes of this course, much of my preparations come from the latter portion. Memory isn’t my main specialty, so I’ve been spending a lot of time reading up on it. Wandering into a relatively new field is always an interesting experience, and on that front I consider myself fortunate: I have a theoretical guide to help me think about and understand the research I’m encountering – evolution. Rather than just viewing the field of memory as a disparate collection of facts and findings, evolutionary theory allows me to better synthesize and explain, in a satisfying way, all these novel (to me) findings and tie them to one another. It strikes me as unfortunate that, as with much of psychology, there appears to be a distinct lack of evolutionary theorizing on matters of learning and memory, at least as far as the materials I’ve come across would suggest. That’s not to say there has been none (indeed, I’ve written about some before), but rather that there certainly doesn’t seem to have been enough. It’s not the foundation of the field, as it should be. 

“How important could a solid foundation really be?”

To demonstrate what I’m talking about, I wanted to consider an effect I came across during my reading: the generation effect in memory. In this case, generation refers not to a particular age group (e.g., people in my generation), but rather to the creation of information, as in to generate. The finding itself – which appears to replicate well – is that, if you give people a memory task, they tend to be better at remembering information they generated themselves, relative to remembering information that was generated for them. To run through a simple example, imagine I was trying to get you to remember the word “bat.” On the one hand, I could just have the word pop up on a screen, tell you to read and remember it. On the other hand, I could give you a different word, say, “cat” and ask you to come up with a word that rhymes with “cat” that can complete the blanks in “B _ _.” Rather than my telling you the word “bat,” then, you would generate the word on your own (even if the task nudges you towards generating it rather strongly). As it turns out, you should have a slight memory advantage for the words you generated, relative to the words you were just given.

Now that’s a neat finding at all – likely one that people would read about and thoughtfully nod their head in agreement – but we want to explain it: why is memory better for words you generate? On that front, the textbook I was using was of no use, offering nothing beyond the name of the effect and a handful of examples. If you’re trying to understand the finding – much less explain it to a class full of students – you’ll be on your own. Textbooks are always incomplete, though, so I turned to some of the referenced source material to see how the researchers in the field were thinking about it. These papers seemed to predominately focus on how information was being processed, but not necessarily on why it was being processed that way. As such, I wanted to advance a little bit of speculation on how an evolutionary approach could help inform our understanding of the finding (I say could because this is not the only possible answer to the question one could derive from evolutionary theory; what I hope to focus on is the approach to answering the question, rather than the specific answer I will float. Too often people can talk about an evolutionary hypothesis that was wrong as a reflection of the field, neglecting that how an issue was thought through is a somewhat separate matter from the answer that eventually got produced).

To explain the generation effect I want to first take it out of an experimental setting and into a more naturalistic one. That is, rather than figuring out why people can remember arbitrary words they generated better than ones they just read, let’s think about why people might have a better memory for information they’ve created in general, relative to information they heard. The initial point to make on that front is that our memory systems will only retain a (very) limited amount of the information we encounter. The reason for this, I suspect, is that if we retained too much information, cognitively sorting through it for the most useful pieces of information would be less efficient, relative to a case where only the most useful information was retained in the first place. You don’t want a memory (which is metabolically costly to maintain) chock-full of pointless information, like what color shirt your friend wore when you hung out 3 years ago. As such, we ought to expect that we have a better memory for events or facts that carry adaptively-relevant consequences.

“Yearbooks; helping you remember pointless things your brain would otherwise forget”

Might information you generate carry different consequences than information you just hear about? I think there’s a solid case to be made that, at least socially, this can be true. In a quick example, consider the theory of evolution itself. This idea is generally considered to be one the better ones people (collectively) have had. Accordingly, it is perhaps unsurprising that most everyone knows the name of the man who generated this idea: Charles Darwin. Contrast Darwin with someone like me: I happen to know a lot about evolutionary theory and that does grant me some amount of social prestige within some circles. However, knowing a lot about evolutionary theory does not afford me anywhere near the amount of social acclaim that Darwin receives. There are reasons we should expect this state of affairs to hold as well, such as that generating an idea can signal more about one’s cognitive talents than simply memorizing it does. Whatever the reasons for this, however, if ideas you generate carry greater social benefits, our memory systems should attend to them more vigilantly; better to not forget that brilliant idea you had than the one someone else did.

Following this line of reasoning, we could also predict that there would be circumstances in which information you generated is recalled less-readily than if you had just read about it: specifically, in cases when the information would carry social costs for the person who generated it.

Imagine, for instance, that you’re a person who is trying to think up reasons to support your pet theory (call that theory A). Initially, your memory for that reasoning might be better if you think you’ve come up with an argument yourself than if you had read about someone else who put forth that same idea. However, it later turns out that a different theory (call that theory B) ends up saying your theory is wrong and, worse yet, theory B is also better supported and widely-accepted. At that point, you might actually observe that the person’s memory for the initial information supporting theory A is worse if they generated those reasons themselves, as that reflects more negatively on them than if they had just read about someone else being wrong (and memory would be worse, in this case, because you don’t want to advertise the fact that you were wrong to others, while you might care less about talking about why someone who wasn’t you was wrong).

In short, people might selectively forget potentially embarrassing information they generated but was wrong, relative to times they read about someone else being wrong. Indeed, this might be why it’s said truth passes through three stages: ridicule, opposition, and acceptance. This can be roughly translated to someone saying of a new idea, “That’s silly,” to, “That’s dangerous,” to, “That’s what I’ve said all along.” This is difficult to test, for sure, but it’s a possibility worth mulling over.

How you should feel reading over old things you forgot you wrote

With the general theory described, we can now try and apply that line of thinking back into the unnatural environment of memory research labs in universities. One study I came across (deWinstanley & Bjork, 1997) claims that the generation effect doesn’t always have an advantage over reading information. In their first experiment, the researchers had conditions where participants would either read cue-word pairs (like “juice” – “orange”, and, “sweet” – “pineapple”) or read a cue and then generate a word (e.g., “juice” – “or_n_ _”). The participants would later be tested on how many of the target words (the second one in the pair) they could recall. When participants were just told there would be a recall task later, but not the nature of that test, the generate group had a memory advantage. However, when both groups were told to focus on the relationship between the targets (such as them all being fruits), the read group’s ability to remember now matched that of the generate group.

In their second experiment, the researchers then changed the nature of the memory task: instead of asking participants to just freely recall the target words, they would be given the cue word and asked to recall the associated target (e.g., they see “juice” and need to remember “orange”). In this case, when participants were instructed to focus on the relationship between the cue and the target, it was the read participants with the memory advantage; not the generate group.

One might explain these findings within this framework I discussed as follows: in the first experiment, participants in the “read” condition were actually also in an implicit generate condition; they were being asked to generate a relationship between the targets to be remembered and, as such, their performance improved on the associated memory task. By contrast, in the second experiment, participants in the read condition were still in the implicit “generate” condition: being asked to generate connections between the cues and targets. However, those in the explicit generate condition were only generating the targets; not their cues. As such, it’s possible participants tended to selectively attend to the information they had created over the information they did not. Put simply, the generate participant’s ability to better recall the words they created was interfering with their ability to remember their associations with the words they did not create. Their memory systems were focusing on the former over the latter.

A more memorable meal than one you go out and buy

If one wanted to increase the performance of those in the explicit generate condition for experiment two, then, all a researcher might have to do would be to get their participants to generate both the cue and the target. In that instance, the participants should feel more personally responsible for the connections – it should reflect on them more personally – and, accordingly, remember them better. 

Now whether that answers I put forth get it all the way (or even partially) right is besides the point. It’s possible that the predictions I’ve made here are completely wrong. It’s just that what I have been noticing is that words like “adaptive” and “relevance” are all but absent from this book (and papers) on memory. As I hope this post (and my last one) illustrates, evolutionary theory can help guide our thinking to areas it might not otherwise reach, allowing us to more efficiently think up profitable avenues for understanding existing research and creating future projects. It doesn’t hurt that it helps students understand the material better, either.

References: deWinstanley, P. & Bjork, E. (1997). Processing instructions and the generation effect: a test of the multifactor transfer-appropriate processing theory. Memory, 5, 401-421.

Within the world of psychology research, time is often not kind to empirical findings. This unkindness was highlighted recently in the results of the reproducibility project, which found that the majority of psychological findings tested did not appear to replicate particularly well. There are a number of reasons this happens, including that psychological research tends to be conducted rather atheoretically (allowing large numbers of politically-motivated or implausible hypotheses to be successfully floated), and that researchers have the freedom to analyze their data in rather creative ways (allowing them to find evidence of effects where none actually exist). These practices are engaged in because positive findings tend to be published more often than null results. In fact, even if the researchers do everything right, that’s still not a guarantee of repeatable results; sometimes people just get lucky with their data. Accordingly, it is a fairly common occurrence for me to revisit some research I learned about during my early psychology education only to find out that things are not quite as straightforward or sensible as they had been presented to be. I’m happy to report that today is (sort of) one of those days. The topic in question has been called a few different things, but for my present purposes I will be referring to it as choice overload: the idea that having access to too many choices actually results in making decisions more difficult and less satisfying. In fact, if too many options are presented, people might even avoid making a decision altogether. What a fascinating idea.

Here’s to hoping time is kind to it…

The first time I had heard of this phenomenon, it was in the context of exotic jams. The summary of the research goes as follows: Iyengar & Lepper (2000) set up shop in a grocery store, creating a tasting booth for either six or 24 varieties of jams (from which the more-standard flavors, like strawberry, were removed). Shoppers were invited to stop by the booths, try as many of the jams as they wanted, given a $1 off coupon for that brand’s jam, and then left. The table with the more extensive variety did attract more customers (60% of those who walked by), relative to the table with fewer selections (40%), suggesting that the availability of more options was, at least initially, appealing to people. Curiously, however, there was no difference between the average number of jams sampled: whether the table had 6 flavors or 24, people only sampled about 1.5 of them, on average, and apparently, no one ever sampled more than two flavors (maybe they didn’t want to see rude or selfish). More interestingly still, because the customers were given coupons, their purchases could be tracked. Of those who stopped at the table with only six flavors, about 30% ended up later purchasing jam; when the table had 24 flavors, a mere 3% of customers ended up buying one.

There are a couple of potential issues with this study, of course, owing to its naturalistic design; issues which were noted by the authors. For instance, it is possible that people who were fairly uninterested in buying jam might have been attracted to the 24-flavor booth nevertheless, simply out of curiosity, whereas those with a greater interest in buying jams would have remained interested in sampling them even when a smaller number of options existed. To try and get around these issues, Iyengar & Lepper (2000) designed another two experiments, one of which I wanted to cover. This other experiment was carried out in a more standard lab setting (to help avoid some of the possible issues with the jam results) and involved tasting chocolate. There were three groups of participants in this case: the first group (n = 33) got to select and sample a chocolate from an array of six possible options, the second group (n = 34) got to select and sample a chocolate from an array of 30 possible options, and a final group (n = 67) were randomly assigned to test a chocolate they had not selected. In the interests of minimizing people’s familiar preferences for such things, only those who enjoyed chocolate, but did not have experience with that particular brand were selected for the study. After filling out a few survey items and completing the sampling task, the participants were presented with their payment option: either $5 in cash, or a box of chocolates from that brand worth $5. 

In accordance with the previous findings, participants who selected from 30 different options were somewhat more likely to say they had been presented with “too many” options (M = 4.88) compared with those who old had 6 possible choices (M = 3.61, on a seven-point scale, ranging from “too few” choices at 1, to “too many” choices at 7). Despite the subjects in the extensive-choice group saying that making a decision as to which chocolate to sample was more difficult, however, there was no correlation between how difficult participants found the decision and how much they reported enjoying making it. It seemed people could enjoy making more difficult choices. Additionally, participants in the limited-choice group were more satisfied with their choice (M = 6.28) than those in the extensive-choice group (M = 5.46), who were in turn more satisfied than those in the no-choice group (M = 4.92). Of particular interest are the compensation findings: those in the limited-choice group were more likely to accept a box of chocolate in lieu of cash (48%) than those in either the extensive-choice (12%) or no-choice conditions (10%). It seems that having some options was preferable to having no options, but having too many options seemed to cause people difficulty in making decisions. The research concluded that, to use the term, people could be overloaded by choices, hindering their decision making process.

“If it can’t be settled via coin flip, I’m not interested”

While such findings are indeed quite interesting, there is no guarantee they will hold up over time; as I mentioned initially, lots of research fails to do likewise. This is where meta-analyses can help. This is the kind of research where the results from many different studies can be examined jointly. Scheibehenne et al (2010) set out to conduct one of their own on the research surrounding choice overload, noting that some of the research on the phenomenon does not point in the same direction. They note a few examples, such as field research in which reducing the number of available items resulted in decreases or no changes to sales, rather than what should have been a predicted uptick in them. Indeed, the lead author also reports that their own attempt at replicating the jam study for their dissertation in 2008 failed, as well as the second author’s attempt to replicate the chocolate experiment. These failures to replicate the original research might indicate that the initial results of choice overload were something of a fluke, and so a wider swath of research needs to be examined to determine if that’s the case.

Towards this end, Scheibehenne et al (2010) collected 50 experiments from the literature on the subject, representing about 5,000 participants in 13 published and 16 unpublished papers from 2000-2009. In total, the average estimated effect size for the choice overload effect across all the experiments was a mere D = 0.02; the effect was all but non-existent. Further analysis revealed that the difference in effect sizes between studies did not seem to be randomly distributed; there were likely relevant differences between these papers determining what kind of results they found. To examine this issue further, Scheibehenne et al (2010) began by trimming off the 6 largest effects from both the top and the bottom ends of the reported research. The results showed that, in the trimmed data set, there was little evidence of difference between the remaining research. This suggests that most of the differences between these studies was being driven by unusually large positive and negative effects.

Returning to the complete, untrimmed data set, Scheibehenne et al (2010) started to pick apart how several moderating variables might be affecting the reported results. In line with the intuitions of Iyengar & Lepper (2000), preexisting preferences or expertise did indeed have an effect on the choice overload issue: people with existing preferences were not as troubled by additional items when making a choice, relative to those without such preferences. However, there was also an effect of publication – such that published papers were somewhat more likely to report an effect of choice overload, relative to unpublished ones – as well as a small effect of year – such that papers published more recently were a bit less likely to report choice overloading effects. In sum, the results of the meta-analysis indicated that the average effect size of choice overload was nearly zero, that older studies which saw publication report larger effects than those that came later or were not published, and that well-defined, preexisting preferences likely remove the negative effects of having too many options (to the extent they actually existed in the first place). Crucially, what should have been an important variable – the number of different options participants were presented with on the high end – explained essentially none of the variance. That is to say that 18 times didn’t seem to make any difference, compared to 30 items or more

“Well, there are too many different chip options; guess I’ll just starve”

While this does not rule out choice overload as being a real thing, it does cast doubt on the phenomenon being as pervasive or important as some might have given it credit for. Instead, it appears probable that such choice effects might be limited to particular contexts, assuming they reliably exist in the first place. Such contexts might include how easily the products can be compared to one another (i.e., it’s harder to decide when faced with two equally attractive, but quite distinct options), or whether people are able to use mental shortcuts (known as heuristics) to rapidly whittle down the number of options they actually consider (so as to avoid spending too much time making fairly unimportant choices). While future examination would be required to test some of these ideas, the larger message here extends beyond the choice overload literature to most of psychology research: it is probably fair to assume that, as things currently stand, the first thing you hear about the existence or importance of an effect will likely not resemble the last thing you do.

References: Iyengar, S. & Lepper, M. (2000). When choice is demotivating: Can one desire too much of a good thing? Journal of Personality & Social Psychology, 79, 995-1006.

Scheibehenne, B., Greifeneder, R., & Todd, P. (2010). Can there ever be too many options? A meta-analytic review of choice overload. Journal of Consumer Research, 37, 409-424.