Maybe It’s Not The Money; Maybe It’s What Money Represents

‘Thank for all for being here. Now pay me”

I’m a big believer in the value of education, which is why I’ve spent so much time educating people (in forums other than here and about topics other than psychology as of late, but I’m always scratching that same itch). As anyone who has been through an education system can tell you, however, not all educators provide the same amount of value. Some teachers and professors have inspired me to reach for new heights while others have killed any interest in a subject I might have had. Some taught me valuable and useful information while others have provided active misinformation. Naturally, if we have the option, we’d all prefer the former type of teacher – the good ones. The same holds true for most parents as well: given the option, they’d prefer their children had access to the best teachers over the worst ones, all else being equal. This is all working under the assumption that good teachers provide better opportunities for their students in the future. I don’t think we’re breaking any new ground here with these premises and I think they’re all sound. This drives students and parents to seek out the best teachers they can find.

Quantifying someone’s quality as an educator is difficult, however. This leads people to fall back on the things they can measure more easily as proxies for educator quality, like student outcomes. After all, if a student cannot perform tasks related to what they were just taught, that’s a reasonable indication that the teacher might not be great at their job. If only matters were that simple we’d have better teachers. They aren’t, though, since such a measure conflates student quality with teaching quality. Put the best teacher in a room of students with an IQ below 80 and you’ll see worse outcomes in terms of student performance than a poor teacher instructing a class with an IQ above 120. Teachers can help you reach for the stars; they just can’t bring the stars to you.

Nevertheless, people do use student outcomes as a proxy for education quality and, as it turns out, students at private schools tend to outperform those at public ones. With limited information available, many people might come to believe that private schools give their children a better education and invest large amounts of resources to ensure their children go there. Perhaps we could improve student performance if we could just send more children to private schools. It’s an interesting suggestion.

“No poor people allowed…until now”

Let’s get the most important question out there first: why would we expect that a private education is better than a public education? The reason this question matters is because the primary difference between these two sources of education is simply the source of funding: private education is funded privately; public education publicly. One might wonder what the source of the funding has to do with the quality of education received, and rightly so. As far as I can tell, that answer should be that funding source per se is largely irrelevant. If you’re buying a new phone, the quality of phone you receive shouldn’t be expected to change on the basis of whether you’re using your money or the government’s money to make the purchase. The same should hold true of education.

As such, if you’re wondering whether private or public education is better, you’re not really looking at the right variables. Whatever factors are important for a good education – class sizes, instructor quality, instruction method, and so on – should be the same for both domains. So perhaps, then, private educations are better because more money allows people to purchase better teachers with better supplies and better methods. As the old saying goes, “you get what you pay for.” Presumably, this would result in children at private schools achieving more in terms of learning and outperforming their public-schooled peers. It might also mean that if public schools just received more money to purchase more materials, space, or better teachers, you’d see student performance begin to increase

That said, this logic usually only holds true to a point. There are diminishing returns on the amount of quality you receive per extra dollar spent. A $5 shirt might be of lower quality than a $30 shirt, but is that shirt six-times better? Is that $120 designer shirt four-times better still? At some point, spending more doesn’t necessarily get you much in the way of a better product.

Hey you tried paying even more than that?

This brings us nicely to the present paper by Pinata & Ansari (2018) who examined a sample of approximately 1100 children’s education-related achievements over time (from birth to age 15). While the paper isn’t experimental in nature, the authors sought to determine to what extent children’s enrollment in private schools affected their performance, as records on their school attendance were available (among other measures). Whether these children attended any private school (yes/no) as well as how much private school they attended were used to predict their ninth-grade performance on a number of standard metrics. These included cognitive, literary, and math skills, as well as working memory abilities. Just to be thorough, they also asked these children how competent they felt in a couple academic domains. The authors also assessed children’s behavioral problems – internal and external – and social skills to see if private school had an impact on those as well. Finally, a number of family variables were collected, including factors like birth weight, maternal employment and vocabulary, and race. In other words, factors unrelated to the public vs private schooling itself.

Turning to the results, when the authors were just trying to predict cognitive and academic performance from the amount of private school attended, there was a noticeable difference. Children who attended any private school tended to outperform those who only attended public school on most of the measured variables. The authors then conducted the same analysis, adding in some of those pesky family variables – like family income – which ended up reducing just about all of those relationships to non-significance, and this was true regardless of how long the children had attended private institutions. In other words, children who attend private school tended to do better than those who attended public school, but this might have very little to do with the schools per se.

While that finding might be interesting to some for reasons related to their finances, it interests me for a different reason. Specifically, at no point in the paper (or the comments/reactions to it) do the authors mention that maybe the difference in performance has to do with some kind of biologically-inherited potential. The ability to learn, like all things biological, is partially inherited. Smart parents tend to have smart children, just like tall parents tend to have tall children. Instead the focus on this paper (and the commentary) seems to revolve predominantly around controlling for the monetary factors.  

Let’s just print more money until everyone’s a genius

Maybe richer parents are able to provide things that poorer parents cannot, and those things lead to better academic performance. Perhaps that’s true, but it does seem to gloss over a rather important fact: wealth is not distributed randomly. Those who are able to achieve higher incomes tend to do so because they possess certain skills that those who fail to achieve high income lack. These could be related to intelligence (factors like good working memories and high IQ) or personality (higher in agreeableness, conscientiousness, or other important factors). This is a long-winded way of saying that people who can successfully complete complicated jobs and show up consistently probably out earn those who mess up everything they touch, frequently miss work, or become distracted by other goals regularly. Each group also tends to have children which inherit these tendencies.

We might expect, then, that parents who have lots of money to spend on an expensive private education are higher-performers, on average; that’s why they have so much extra cash and value spending it on what they think is a good education. They’re also the same kind of parents who are likely to have children who are higher performers, because the children genetically resemble them. This would certainly explain the present set of findings.

When people have different biological performance ceilings the best teachers might help students reach those ceilings without changing where they reside. Past a certain point, then, educator quality may fail to have a noticeable effect. Let’s put that in a sports example: a great coach may make his players as good as they can be at basketball and as a team working together, but he can’t coach them into being taller. No amount of money can buy that ability in a coach. Conversely, some people are likely to succeed even despite a poor education simply because they’re capable enough on their own that they don’t need much additional guidance. A poor teacher to them is simply white noise in the background they can ignore as they achieve all on their own.

“Can you please shut up so I can get back to being great?”

All of this is not to say that educators don’t vary in quality, but it could be the case that the distribution of that quality is at least partially (perhaps even largely or entirely) independent of money at the moment. Maybe teachers are being hired on the basis of things that have little to do with their ability to provide quality education. In higher education this is most certainly the case, where publications and the ability to bring in grant money look appealing.

There is also the lurking matter of how peer quality influences the education of other students. A healthy portion of school life for any child involves managing the social world they attend school in. Children transferring into one school from another – private or public – find themselves faced with the prospect of navigating a new social hierarchy, and that goal tends to distract from education. Similarly, children who find themselves in a school where their peers don’t value education may not put learning at the top of their to-do list, as it affords them little social mobility (at least in the short term). It’s also possible that even poor-performing children will find little motivation to improve when you surround them by high-performing children if the gap between them is too wide. Since they can’t improve enough to see social gains from it, they may disengage from education and pursue other goals.

It’s not like the only thing that can change between schools – public or private – is educator quality or the amount of money they have for books. Many other moving parts are at work, so simply shuffling more children into private schools shouldn’t be expected to just improve outcomes.

References: Pinata, R. & Ansari, A. (2018). Does attendance in private schools predict student outcomes at age 15? Evidence from a longitudinal study. Educational Research, DOI: 10.3102/0013189X18785632

Unusual Names In Learning Research

Learning new skills and bodies of knowledge takes time, repetition, and sustained effort. It’s a rare thing indeed for people to learn even simple skills or bodies of knowledge fluently with only a single exposure to them if they’re properly motivated. Given the importance of learning to succeed in life, a healthy body of literature in psychology examines people’s ability to learn and remember information. This literature extends both to how we learn successfully and the contexts in which we fail. Good research in this realm will often leverage something in the way of adaptive function for understanding why we learn what we do. It is unfortunate that this theoretical foundation appears to be lacking in much of the research on psychology in general, with learning and memory research being no exception. In the course I taught on the topic last semester, for instance, I’m not entirely sure the world “relevance” appeared once in the textbook I was using to help the reader understand our memory mechanisms. There was, however, a number of parts of that book which caught my attention, though not for the best reasons.

You have my attention, but no longer have a working car.

Recently, for instance, I came upon a reference to a phenomenon called the labor-in-vain effect through this textbook. In it, the effect was summarized as such: 

Here’s the basic methodology. Nelson and Leonesio (1988) asked participants to study words paired with nonsense syllables (e.g., monkey–DAX). Participants made judgments of learning in an initial stage. Then, when given a chance to study the items again, each participant could choose the amount of time to study for each item. Finally, in a cued recall test, participants were given the English word and asked to recall the nonsense syllable….Even though they spent most of their time studying the difficult items, they were still better at remembering the easy ones. For this reason, Nelson and Leonesio labeled the effect labor in vain because their experiment showed that participants were unable to compensate for the difficulty of those items

As I like to be thorough when preparing the materials for my course, I did what every self-respecting teacher should do (even though not all of them will): I went to go track down and read the primary literature upon which this passage was based. Professors (or anyone who wants to talk about these findings) ought to go read the source material themselves for two reasons: first, because you want to be an expert in the material you’re teaching your students about (why else would they be listening to you?) and, second, because textbooks – really secondary sources in general – have a bad habit of getting details wrong. What I found in this case was not only that the textbook mischaracterized the effect and failed to provide crucial details about the research, but the original study itself was a bit ambitious in their naming and assessment of the phenomenon. Let’s take those points in order.

First, to see why the textbook’s description wasn’t on point, let’s consider the research itself (Nelson & Leonesio, 1988). The general procedure in their experiments was as follows: participants (i.e., undergraduate students looking for extra credit) were given lists to study. In the first experiment these were trigrams (like BUG or DAX), in the second they were words paired with trigrams (like Monkey-DAX), and in the third they were tested on general-information questions they had failed to answer correctly (like, “what is the capital of Chile?”). During each experiment, the participants would be broken up into groups that either emphasized speed or accuracy in learning. Both groups were told they could study the target information at their own pace and that the goal was to remember as much of the information as possible, but the speed groups were told their study time would count against their eventual score. Following that study phase, participants were then given a recall task after a brief delay to see how successful their study time had been. 

As one might expect, the speed-emphasis groups studied the information for less time than the accuracy-emphasis groups. Crucially, the extra study time invested by the participants did not yield statistically significant gains in their ability to subsequently recall the information in 2 of the 3 experiments (in experiment three, the difference was significant). This was dubbed the labor-in-vain effect because participants were putting in extra labor for effectively little to no gain.

We can see from this summary that the textbook’s description of the labor-in-vain effect isn’t quite accurate. The labor in vain effect does not refer to the fact that participants were unable to make up the difference between the easy and hard items (which they actually did in one of the three studies); instead, it refers to the idea that the participants were not gaining anything at all from their extra study time. To quote the original paper: 

We refer to this finding of substantial extra study time yielding little or no gain in recall as the labor-in-vain effect. Although we had anticipated that extra study time might yield diminishing (i.e., negatively accelerated) gains in recall, the present findings are quite extreme in showing not even a reliable gain in recall after more than twice as much extra study time.

This mischaracterization might seem like a minor error speaking to the meticulousness of the author, but that’s not the only problem with the book’s presentation of the information. Specifically, the textbook provided no sense as for the exact methodological details, the associated data, and whether the interpretation of these findings were accurate. So let’s turn to those now.

If the labor will all be in vain, why bother laboring at all?

The general summary of the research I just provided is broadly true, but very important details are missing that help contextualize it. The first of these involves how the study phases of the experiments took place. Let’s just consider the first experiment, as the methods are broadly similar across the three. In the study phase, the participants had 27 trigrams to commit to memory. The participants were seated at a computer, and one of these trigrams would appear on the screen at a time. After the participants felt they had studied it enough, they would hit the enter key to advance to the next item, but they could not go back to previous items once they did. This meant there was no ability to restudy or practice test oneself in advance of the formal test. To be frank, this method of study resembles no kind that I know humans to naturally engage in. Since the context of studying in the experiment is so strange, I would be hesitant to say that it tells us much about how learning occurs in the real word, but the problems get worse than that.

As I mentioned before, these are undergraduate participants trying to earn extra credit. With that mental picture of the samples in mind, we might come to expect that the participants are a little less than motivated to deliver a flawless performance. If they’re anything like the undergraduates I’ve known, they likely just want to get the experiment over and done with so they can go back to doing things they actually want to. In terms of the interests of college students, learning nonsense syllables isn’t high on that list; in fact, I don’t think that task is high on anybody’s list. The practical information value of what they’re learning is nonexistent, and very little is riding on their success. It might come as no surprise, then, that the participants dedicated effectively no time to studying these items. Bear in mind, there were 27 of these trigrams to learn. In the speed group, the average number of seconds devoted to study was 1.9 per trigram. Two whole seconds of learning per bit of nonsense. In the accuracy group, this study time skyrocketed to a substantial…5.4 seconds.

An increase of 3.3 seconds per item does not strike me as anything I’d refer to as labor, even if the amount of study time was nominally over twice as long. A similar pattern emerged in the other two experiments. The speed/accuracy study times were 4.8 and 15.2 in the second study, and 1.2 and 8.4 in the third. Putting this together up to this point, we have (likely unmotivated, undergraduate) participants studying useless information in unnatural ways for very brief periods of time. Given that, why on Earth would anyone expect to find large differences in later recall performance?

Speaking of eventual performance, though, let’s finally consider how well each group performed during the recall task; how much of that laboring was being done in vain. In the first experiment, the speed group recalled 43% of the trigrams; the accuracy group got 49% correct. That extra study time of about 3 seconds per item yields a 6% improvement in performance. The difference wasn’t statistically significant but, again, exactly how large of an improvement should have been expected, given the context? In the second study, these percentages were 49% and 57%, respectively (a gain of 8%); in the third, they were 75% and 83% (another 8% difference that actually was statistically significant given the larger sample size for experiment 3). So, across three studies, we do not see evidence of people laboring in vain; not really. Instead, what we see is that very small amounts of extra time devoted to studying nonsense in unusual ways by people who want to be doing other things yields corresponding small – but consistent – gains in recall performance. It’s not that this labor was in vain; it’s that not much labor was invested in the first place, so the gains were minimal.  

If you want to make serious gains, you’ll need more than baby weight

On a theoretical level, it sure would be strange if people would spend substantially extra time laboring in study to make effectively no gains. Why waste all that valuable time and energy doing something that has no probability of paying off? That’s not something anyone should posit a brain would do if they were using evolutionary theory to guide their thinking. It would be strange to truly observe a labor-in-vain effect in the biological sense of the word. However, given a fuller picture of the methods of the research and the data it uncovered, it doesn’t seem like the name of that effect is particularly apt. The authors of the original paper seem to have tried to make these results sound more exciting than they are (through their naming of the effect and the use of phrases like, “…substantial extra study time,” and differences in study time that are, “highly significant,” as well as an exclamation point here and there). That the primary literature is a little ambitious is one thing, but we also saw that the secondary summary of the research by my textbook was less than thorough or accurate. Anyone reading the textbook would not leave with a good sense for what this research found. It’s not hard to imagine how this example could extend further to a student summarizing the summary they read to someone else, at which point all the information to be gained from the original study is effectively gone.

The key point to take away from this is that textbooks (indeed, secondhand sources in general) should certainly not be used as an end-point for research; they should be used as a tentative beginning to help track down primary literature. However, that primary literature is not always to be taken at face value. Even assuming the original study was well-designed and interpreted properly, it would still only represent a single island of information in the academic ocean. Obtaining true and useful information from that ocean takes time and effort which, unfortunately, you often cannot trust others to do on your behalf. To truly understand the literature, you need to dive into it yourself.

References: Nelson, T. & Leonesio, R. (1988). Allocation of self-paced study time and the “Labor-in-Vain Effect”. Journal of Experimental Psychology, 14, 676-686.