Why do humans and bananas alike get seduced into obsession with the events in Salem, Massachusetts, in 1692? It’s not the mystery of why twenty people were executed, and hundreds imprisoned, for a supernatural crime. It’s not the mystery of why everyone suddenly got tired of hanging witches in a matter of months. It’s not the mystery of why it was the last witch trial of its scale in the Western world. And it’s not the play written by a communist that everyone had to read in high school, which functions as a curiosity repellant.

In fact, as you will quickly learn if you begin to study Salem, the fascination lies in an ancient dispute between two men of Boston, one of whom had no involvement in the Salem witch trials at all. The first, the politically connected minister Cotton Mather, wrote the first history of the Salem witch trials in the immediate aftermath of the event, Wonders of the Invisible World, attempting to whitewash the affair and justify the executions. The second, the humble but successful weaver and merchant Robert Calef, did not much like this, and wrote the second history of the trials a few years later, More Wonders of the Invisible World. They came to very different conclusions.

Ever since, it has been a tradition for Salemheads to take a side in this dispute. The side you choose is important, because it informs all the decisions you will arrive at about the import of each piece of evidence. Your judgments about every question arising from the tragedy will have curious relevance to the matter of which man you back in this antique beef.

Say you take Cotton Mather’s part, for example. Mather argued that demonic forces were assaulting Massachusetts, beginning years before the Salem tragedy (especially in his 1689 book Memorable Providences). In his history of the Salem witch trials (and for the rest of his life) he attempted to vindicate the judges and accusers, and insisted that the condemned witches were, in fact, guilty. In this case, you might find it convenient to argue that the witch belief was ubiquitous—or even, as some scholars have gone so far as to argue, that witchcraft was actually real, and that the executed witches really deserved to die. You might be tempted to ignore or reinterpret evidence that makes Cotton Mather look like a fool.

Robert Calef, on the other hand, argued that the supposed witchcraft was never real, that the condemned were innocent, that the accusers were frauds, and that a great injustice took place. If you take his part, you might emphasize the doubters who dared to express their doubts at the time and long before, and acknowledge that witchcraft isn’t real. You are free to argue that the tragedy at Salem was the result of solely human agency, including the agency of moral entrepreneurs. And you will delight in evidence that makes Cotton Mather look like a fool, especially if he wrote it himself.

This illustrates the problem with trying to tell just a little piece of the history of the Salem witch trials: everything is connected to everything else. Nevertheless, my hope with this introduction is to tell a small but critically important part of the story, and to gesture at how it connects with all the rest. For me, as a Calef partisan, that small piece is A Brand Pluck’d Out of the Burning, Cotton Mather’s narrative of Mercy Short. It was written in 1693, circulated among his friends, lost to history, and rediscovered hidden away in some dusty desk almost two centuries later. It was first published in complete form by George Lincoln Burr in his 1914 compendium of Salem sources, Narratives of the Witchcraft Cases, 1648-1706, and you can read it for free (it starts on page 255).

I choose this text not just because it is very interesting and funny and makes Cotton Mather look like a fool, but because it highlights one of the most important and frequently neglected aspects of the events: the fraud, deceit, magic tricks, and pranks conducted by the accusers. We get a more detailed picture here than in any other case study.

If you’ve heard one thing about the witch trials, you’ve probably heard of the theory that the whole tragedy was caused by ergot poisoning from moldy grain. This is part of a genre of medical excuses offered for the accusers, ranging from ergot, Lyme disease, or encephalitis on the one hand, to more psychological but still medical diagnoses like “hysteria” (or the more modern “conversion disorder”) on the other. The latter is a way for fans of the spooky and mysterious to preserve their prescientific world model while pretending to be scientific. The main point of my argument is that there was in fact nothing spooky or mysterious about the events in Salem. Neither demonic nor psychological explanations are indicated.

For many behaviors of the accusers, it may be difficult to convincingly distinguish innocent psychological causes (in the vein of “hysteria”) from conscious fraud. For other behaviors, however, conscious fraud is the only conclusion—specifically, behaviors requiring premeditation, practice, intentional deception, and/or collusion with confederates. Fainting and convulsing might be argued to be symptoms of hysteria (even the peculiar pattern exhibited by the accusers, which would conclusively rule out non-psychological medical causes), but performing magic tricks or elaborate group pranks must be ruled conscious fraud. That is why the narrative of Mercy Short is so important.

The Sport of Mercy Short

When Mercy Short was about fourteen years old, her parents were killed by the French and Indians, and she was taken captive, along with her brothers and sisters. They were taken to Canada, and about eight months later these hostages were “redeemed” and brought to Boston. Even if you find her subsequent behavior to be very naughty, you must admit that she had been through a great deal of trauma. Her behavior was also, with exceptions, extremely funny, and to my knowledge she did not actually get anyone judicially murdered, unlike the accusers at Salem.

In the summer of 1692, when Mercy Short was probably sixteen and the Salem witch trials were in full swing, she first displayed the by-then-stereotypical symptoms of being bewitched. Mercy, who was working as a servant in Boston, was sent by her mistress on an errand to the Boston jail, where the accused witches of Salem were being held. There, she supposedly encountered Sarah Good (Cotton Mather calls her “the Hag”) who, along with her four-year-old daughter, was imprisoned as a suspected witch. Goodwife Good asked Mercy for some tobacco, and Mercy threw a “handful of shavings” at her in answer, saying “that’s tobacco good enough for you!” Goodwife Good bestowed some choice words upon her, and that’s when Mercy Short first presented with the “fits” of the bewitched. History does not tell if she met anyone else on her errand to the prison, though we will see that she had an extensive social network.

Mercy spent a few weeks having fits and “fasting” for up to twelve days (the mystery of her fasts will soon be explained), but she got tired of it and went back to normal until November of 1692, about a month after the witch trials fell apart and the Court of Oyer and Terminer (the special witch court) was dissolved. Around November 22 she began to have fits again, and fasted for nine days, but again got tired of it, at least for three days.

At the end of those three days, though, around December 4th, she attended Cotton Mather’s church and made a terrible scene, pretending to see spirits around her and again having fits. The men of the church carried her, probably kicking and screaming, to the house of the nearest neighbor who would take her in, and there she stayed for several more weeks. She repeated the cycle at least one more time, getting better for a while and then having another tantrum in church, until her final deliverance on March 16, 1693. (For those counting, that is four cycles of possession over about ten months.)

Now we will see all the tricks that she got up to during her episodes of spirit possession, which Cotton Mather admirably records in enough detail for us to interpret what was really going on. His fans are correct in finding him to be an excellent and detailed observer, at least for facts that he thought would improve his case.

Cotton Mather notes that up to “half an hundred” people would be present to watch Mercy Short perform.

Sleight of Hand

Cotton Mather reports that Mercy Short could make solid objects disappear, and in some cases, reappear. For example, she picked his pocket and relieved him of a gold ring, then notified him that the spirits had taken it. He was amazed, and even more amazed when she told him where to find it—and indeed, there it was!

Shee fell a Laughing at One in the Room [note: this is how Cotton Mather talks about himself], and asked him, Whether hee had not a Gold Ring about him? Hee knew hee had, and look’d for it in the pocket where hee knew it was, but it was missing; and Shee, laughing, told him, That a Spectre had newly taken it out; but, said shee, “Look in such a Place and you shall find it.” Accordingly, hee Look’d and Found. Shee added, “They said, that if hee putt it on, They would have it off his Finger again before hee gott home.” Hee, to spite Them, Try’d; but tho’ hee diverse times between That and Home, thought his Finger taken with an odd Numbness, yett hee kept it on.

One of the explanations for the long fasts that so amazed Cotton Mather is simply that she did in fact eat and drink the whole time, which Cotton Mather admits, although he minimizes the amount: “One Raw Pear shee ate, and now and then an Apple, and some Hard Cider shee drank, things that would rather sett an Edge upon the Severity of her Fast: Sometimes also a Chestnut might go down into her Craving Stomach and sometimes a little Cold Water.”

But the more interesting explanation is revealed by Cotton Mather as another miracle: when she was offered food—biscuits or apples, perhaps—the “spirits” would snatch the treats out of her hand (that is, she would make them “disappear” into her sleeve by sleight of hand). Then she would complain that the spirits were eating them right in front of her, which of course no one else could see. To Cotton Mather’s credulous eyes, the food had disappeared and been eaten by specters. In reality, presumably she just saved it for later when she could eat without being observed.

The Impudence of the Troublesome Spectres was now somewhat more Daring and Broad-faced than formerly. It grew common with them to snatch from her such Apples and Biskets, as were given her to Try whether shee could eat them; so that no more could ever bee seen of them. And Mercy Short affirm’d, That shee saw the Spectres (tho’ wee could not,) eating them in the Room, what wee perceived they had stolen from her.

(Yet another explanation for the fasts is that she had confederates who could be relied on to bring her whatever she required in secret, as we shall see in the next section.)

Although a less impressive trick, Mercy Short also engaged in the practice of sticking pins into herself and saying that the specters had done it. This was a relatively common practice of the bewitched, both at Salem and across the ocean in England and Scotland. Sometimes she would prick herself and display the marks; other times she would leave the pins in for spectators to see. Sometimes she would sneak pins into her mouth and pretend that the spirits were forcing her to swallow them, so that the people hanging around her bedside (such as Cotton Mather) would have to pry her mouth open to remove them. Amazingly, they always succeeded in doing so before she could be forced by the specters to swallow them:

Besides the Thousands of cruel pinches given her by those Barbarous Visitants, they stuck innumerable pins into her. Many of those pins They did themselves pluck out again; and yett They left the Bloody Marks of them, which would bee as tis the strange Property of most Witch-wounds to bee, cured, perhaps in less than a Minute. But some of the Pins They left in her, and those wee took out, with Wonderment. Yea, sometimes They would force Pins into her Mouth, for her to swallow them; and tho’ Shee strove all shee was able to keep them out, yett They were too hard for her. Only before they were gott into her Throat, the Standers-by would by some Dexterity gett hold of them, and fetch them away.

Another genre of trick that amazed Cotton Mather was one involving books. For example, in her “trance,” Mercy Short would pick up a Bible, and without seeming to even look at it, would turn a few pages and fold down a leaf. It would just happen to be a verse with direct relevance to her current situation!

In her Trances, a Bible Happening to ly on her Bed, shee has taken it up, and without ever casting her Eye upon it, shee has Turned over many Leaves, at last folding down a Leaf to a Text, I holding up the Text unto the Spectres; but of all the Texts in the Bible, which do you think it was? T’was That, in Rev. 12. 12, The Divel is come down unto you, having great Wrath, because hee knows hee hath but a short Time.

This is a relatively simple trick to pull off, but one that would require preparation (locating appropriate verses in secret and memorizing or marking in order to find them again without seeming to look at the pages). She was happy to repeat this trick over and over, to pick out Bible verses or Psalms for the assembled company to sing. In one case, she even added a spooky bit of information to the trick, saying that the spirits told her the verse she’d found was one Cotton Mather had never preached upon—and it was true!

…and tho’ shee saw not the Text herself, yett Folding down a Leaf unto it, shee held it up unto the Spectres, for Them to Read it; adding withal, That her Name was already in that Book of the Lamb, and therefore it should never come into Their cursed Book! To which They Reply’d, Shee had shown them a Scripture which one (they named) had never yett preached upon: and in That, they spoke True.

Another trick Mercy Short performed was to (pretend to) read from the magical Devil’s book which the specters presented to her over and over. (According to the witch believers, the Devil and his minions would try to convince bewitched people to sign their special book, or even just touch it, to certify a contract with the Devil, which is how they claimed people became witches.) At one point, Mercy spelled out loud a word which is “too hard” for her, “Quadragesima,” and that Cotton Mather imagined she couldn’t possibly know except by reading it from the evil book. And, like the Bible verses she chose without looking, they were curiously relevant! Quadragesima is the first Sunday of Lent, in the Catholic faith which Cotton Mather regarded as diabolical (as we shall see more of later), which is why the demons would be talking about it. I can’t tell when exactly Quadragesima occurred in 1693, but by mentioning the date, Cotton Mather implies that he, at least, believed it to be curiously near in time. (Puritans did not celebrate holidays like Easter or Christmas, but Cotton Mather had a library of Catholic books for hate-reading, as we shall see.)

It particularly surprised some in the Room, on the Eve of March 9, 1693, to overhear her, in the Book then opened unto her, spelling a Word that was too hard for her; but from the best Judgment that could bee made of the Letters that shee recited, it was Quadragesima.

If he wasn’t simply imagining diabolical patterns in gibberish, presumably Mercy Short had overheard this word, or got it from one of her confederates especially for the prank—at least one of whom had access to Mather’s own library. We turn now to the evidence for an army of confederates.

Collusion with Confederates

The most daring prank that Mercy Short pulled off is the matter of the cat in the cockloft. She imperiously informed Cotton Mather that she desired a visit from him together with the governor of Massachusetts, Sir William Phips—and they obliged! When these distinguished gentlemen visited her bedside, she told them that she overheard the specters saying that they had dropped their evil, magical book, the book they used to sign up witches to bind them to the Devil. They dropped it, she told them, in the garret (attic) of a cockloft (an enclosed crawlspace at the very top of an attic) of a “person of quality” in the neighborhood, and if the gentlemen were to hurry they could retrieve the cursed book! They were suspicious that such a book even existed, and did not, in fact, hurry. But a few days later, they got around to sending a servant to check the cockloft—who was greeted by a spooky black cat jumping out over him! A cat, furthermore, never known to be at that house!

After the servant got jump scared, Mercy sent for Cotton Mather, and was very annoyed that he and the governor took so long to look for the book. In that time, she told him, one of the specters turned himself into a cat, and was in the process of retrieving the book when the servant came upon him in the cockloft. She knew all about this even though, Cotton Mather assures us, none of her attendants had told her anything about the cat. In Cotton Mather’s interpretation, she could have only known about it from the specters.

A more realistic interpretation, of course, is that Mercy Short arranged with confederates to lock up a poor cat in said cockloft, hoping to jump scare Cotton Mather and the governor, but succeeding only in scaring a servant. Nevertheless, the trick worked well enough to be recorded as a diabolical miracle. It leads Cotton Mather to some deep metaphysical speculations:

Whether the Cat were what was pretended, I shall give no Opinion: tho’ I know the Assertion of some, That every Spirit is endued with an Innate Power by which it can attract suitable matter out of all Things for a Covering or Body, of a proportionable Form and Nature to itself: which Assertion, Well stated, Proved, and Applyed, would solve some of the hardest Phenomena that belong to the uncouth and horrid Shapes, wherein mischiefs are done by Witchcraft.

Another prank involves interfering with Cotton Mather’s books from afar. Mercy Short told Cotton Mather that the specters had a certain book that they used for their diabolical rites, and whenever they wanted to use it, they would steal it from Cotton Mather’s own library. (Apparently, the book named is an utterly normal Catholic devotional book that Cotton Mather kept for hate-reading, as mentioned above. It was Catholic, so of course the devils loved it—the Puritans absolutely despised Catholics.) Mercy described the book in detail and “saw” the specters fold a page down, and indeed, in his own home, Mather found his own copy with its page folded down by the pesky devils. Not only did she have confederates—she had a mole in Cotton Mather’s own house! (Perhaps it was another servant girl.) She and her confederate repeated the trick again, in the night, after Cotton Mather had carefully checked that no pages were folded:

But that which added unto the surprise was, That hee found a Leaf doubled down in the Book, which hee could not conceive how it should come: and when a Night or Two after, just as hee went unto his Rest, hee left this Book on his Table in his Study, carefully seeing that there should not bee one Leaf at all folded in it; yett the next morning hee found Three Leaves unaccountably Folded, and then Visiting Mercy, hee perceived the Spectres bragging, That tho’ shee had [said] shee would warrant them, that Gentleman would keep his Book out of their Hands, yett they had Last Night stole the Book again unto one of their meetings, and folded sundry Leaves in it. [Again “hee” refers to Mather himself; this is just how hee is.]

She repeated the trick yet again, describing to him another Catholic book in his library that the specters liked to borrow. This time the specters told her they’ve returned it with the back side outwards, and when Cotton Mather checked, what the spirits told her was true! Cotton Mather takes this as evidence that Catholicism was even more diabolical than he’d supposed.

Mercy Short had sources that would tell her gossip, especially about herself, and Mather attributed this to diabolical messengers rather than confederates:

T’was an ordinary thing for the Divel to persecute her with Stories of what this and that Body in the Town spoke against her. The Unjust and Absurd Reflections cast upon her by Rash People in the coffee-houses or elsewhere, Wee discerned that the Divel Reported such Passages unto her in her Fitts, to discourage her.

There are many references in Cotton Mather’s writings to this hated coffee house and its “coffee-house wits,” hinting at the presence of a group of skeptics whose existence vexed him—two of whom may have been Robert Calef and Thomas Brattle (see the first section of the reading list).

Mercy performed long monologues while pretending to address the invisible specters, seeming, in Cotton Mather’s rendition, quite organized in her speech, and not like a rambling “distracted” person. She predicted a fire, and there was a fire; this may simply be coincidence, or a hint that her confederates were up to much worse mischief than we have seen so far. This is a piece of her monologue, with Cotton Mather’s note in parentheses:

Well if You do Burn mee, I had better Burn for an Hour or Two here then in Hell forever.—What? Will you Burn all Boston and shall I bee Burnt in that Fire?—No, tis not in Your Power. I hope God won’t lett you do that. (Memorandum, The Night after these Words were spoken, the Town had like to have been burn’t; but God wonderfully prevented it.)

Mercy reported that the specters were planning to have a Christmas dance in her room, which was doubly shocking, because neither Christmas nor dancing was halal according to the Puritans. And Mather reports that on “the twenty-fifth of December,” which is as close as he will come to mentioning the name of the forbidden holiday, the attendants in Mercy Short’s room did hear (and feel!) a dance. She would not have been able to perform this feat herself, and the description implies more than one hidden confederate to make the required noise:

On the twenty-fifth of December it was, that Mercy said, They were going to have a Dance; and immediately those that were attending her, most plainly Heard and Felt a Dance, as of Barefooted People, upon the Floor; whereof they are willing to make oath before any Lawful Authority.

It’s a common occurrence for Cotton Mather to threaten to provide witness statements on various incredible points, and on one point he actually followed through, as we will see in the next section.

The specters scratched on the bed and on the wall:

At other times there would be heard, it may bee, by more than seven Witnesses at a time, the Scratches of the Spectres on the Bed and on the Wall.

The demons scratched and pricked the visitors, which could either mean that the confederates were among the assembled company pretending to be pricked, or that the confederates were going around scratching and pricking innocent bystanders. The former interpretation is suggested by how fanciful some of the reports of spirit molestation are:

And whether it were from the Mistake or from the Malice of the Spectres, it was no Rare Thing for the Standers-by to have their Arms cruelly scratch’d, and Pins thrust into their Flesh, by these Fiends, while they were molesting of Mercy Short. Yea, several Persons did sometimes actually lay their Hands upon these Fiends; the Wretches were Palpable, while yett they were not Visible, and several of our People, tho’ they Saw nothing, yett Felt a Substance that seem’d like a Cat, or Dog, and tho’ they were not Fanciful, they Dy’d away at the Fright! This Thing was too Sensible and Repeated a Thing, to bee pure Imaginacion.

It should be noted that during this time, the church youth groups, “both sexes apart,” were meeting in Mercy Short’s chambers, providing a potential pool of confederates for her antics.

Often when the specters were moving around the room (according to Mercy Short), some visitors struck at the invisible specters with their swords. This touches on a widespread bit of witchcraft mythos, that a specter of a witch struck in invisible form would bear the wound in real life—a common motif both during the Salem panic and years before and after. But when the visitors who dared to strike at the specters eventually left and made their way home, the specters got their revenge by haunting and scaring them—and Mercy knew about it every time! Again, this could either be explained by her confederates pretending to have been haunted, or by her confederates actually following other visitors to scare them (possibly including Cotton Mather, as we have seen how he refers to himself in the third person in this and other texts he wrote on plausibly-deniable sock accounts, such as his biography of Sir William Phips). A somewhat darker story we will touch on presently hints at the latter.

It was particularly remarkable, That some who were very Busy in this method of treating the Spectres [striking them with swords], upon a presumption that they might bee corporally present, (tho’ covered with such a Cloud of Invisibilitie as Virgil, I remember, gave once unto his Eneas), were terribly scared with Apparitions in their journeyes home, whereof, tho’ they made no manner of Report, yett Mercy Short was presently after able to tell her Attendents; as having heard the Spectres brag unto her, and unto one another, how They had paid such and such for striking at them.

The cruelest prank that Mercy Short and her confederates committed is reported by Cotton Mather in his diary. Mather’s wife gave birth to a severely deformed baby who sadly did not live long. He suspected witchcraft in the matter, he says, because shortly before the birth, his wife was badly frightened by a specter on their porch. Mercy Short knew about it, and claimed the specters bragged about doing it. Which is to say, she was in on it.

I had great Reason to suspect a Witchcraft, in this preternatural Accident; because my Wife, a few weeks before her Deliverance, was affrighted with an horrible Spectre, in our Porch, which Fright caused her Bowels to turn within her; and the Spectres which, both before and after, tormented a young Woman in our Neighbourhood, brag’d of their giving my Wife that Fright, in hopes, they said, of doing Mischief unto her Infant at least, if not unto the Mother…

While we no longer believe that frights during pregnancy cause birth defects, scaring a pregnant woman severely enough to cause her physical distress was a foul and heartless thing for the Mercy Short gang to do, and puts a damper on our appreciation for their cleverness. Robert Calef mentions their “vicious courses” in later years.

While this has not been a complete list of the tricks and deceptions enacted by Mercy Short and her confederates, it is enough to get a strong sense of the proceedings, and of how these activities were interpreted by the minister Cotton Mather. I now turn to the relation of a few tricks pulled off by another Boston girl later that year, Margaret Rule.

As a reminder, Cotton Mather’s narrative of the Salem witch trials was called Wonders of the Invisible World, and his Mercy Short narrative was called A Brand Pluck’d Out of the Burning. His narrative of his next witchcraft case, that of Margaret Rule, would be creatively titled Another Brand Pluck’d Out of the Burning, Or, More Wonders of the Invisible World, a sort of double sequel. When Robert Calef would eventually publish his book as More Wonders of the Invisible World, this was not a title invented by Calef to mock Cotton Mather—it was Mather’s own choice, which makes it even funnier to me. It is available in Burr’s book beginning on page 308 (see the first section of the reading list, below).

The Fools of Margaret Rule

Cotton Mather’s careful observations reveal to us (though seemingly not to himself) the magic tricks, deceptions, and collusions of Mercy Short. Another aspect of the case comes through clearly to us, though apparently not to him—that the townspeople of Boston were getting tired of Mercy Short’s antics. Mather notices that citizens promised to show up to pray and fast for her, but tended to realize at the last minute that they had something more important to do, which he attributes to the work of the Devil:

It was also remarkable that when wee were intending a Day of Prayer, the Spectres would ussually advise her of our Intention, and brag that They would hinder the People from coming: According to which Brag of Theirs, t’was wonderful to see how many Pious Christians that were desirous to have been with us were hindered of their Desires, by unexpected occasions pressing in upon them.

At any rate, a new seventeen-year-old girl, Margaret Rule, entered the stage on September 10th, 1693, a few months after the conclusion of Mercy Short’s performance. Margaret Rule made her entrance in the same manner as Mercy: by having a fit in church. Like Mercy, she was carried home. She took to her bed to perform for onlookers, and especially for Cotton Mather, just as Mercy Short had.

Margaret Rule’s repertoire overlapped substantially with that of Mercy Short. She would pinch herself “black and blew” and stick herself with pins in the neck, back, and arms. She would fast for up to nine days, taking only spoonfuls of rum (at least in front of observers who were not her confederates). There were noises around the room that could only have come from specters. She knew gossip that could only have come from demons. It would get repetitive to list them all.

I will focus here on only two tricks pulled off by Margaret Rule. Both clearly required confederates, and it is possible that the same gang was involved, with different front-women performing the lead role, although there are suggestions that it may have been a rival gang.

The first trick is relayed in Cotton Mather’s diary (relevant passages reprinted in Burr’s book, and also available on page 173 of this edition). He had written up lecture notes for “discourses” that he intended to preach in church at Salem, and also to publish. Unfortunately, his lecture notes were stolen, and there could only be one culprit—Satan.

I had largely written three Discourses, which I designed both to preach at Salem, and hereafter to print. These Notes were before the Sabbath stolen from mee, with such Circumstances, that I am somewhat satisfied, The Spectres, or Agents in the invisible World, were the Robbers.

His suspicion was confirmed when Margaret Rule told him that the specters confessed to stealing his notes, but she consoled him that he’d get them back eventually:

As for my missing Notes, the possessed young Woman, of her own Accord, enquir’d whether I missed them not? Shee told mee, the Spectres brag’d in her hearing, that they had rob’t mee of them; shee added, Bee n’t concern’d; for they confess, they can’t keep them alwayes from you; you shall have them all brought you again.

As the final act of the trick, he found his notes again—completely preserved, but scattered around the streets of Lynn, Massachusetts, a few miles from Boston.

On the fifth of October following, every Leaf of my Notes again came into my Hands, tho’ they were in eighteen separate Quarters of Sheets. They were found drop’t here and there, about the Streets of Lyn; but how they came to bee so drop’t I cannot imagine; and I as much wonder at the Exactness of their Preservation.

Apparently, Margaret’s gang had returned to the tradition of lighthearted prank content. They stole Mather’s notes surreptitiously, let him know the devils had taken them, then returned them to him under circumstances guaranteed to amaze him. (However, her specters also bragged to her about stealing someone’s will, which was in fact stolen, and there’s no indication that it was returned.)

But Margaret Rule’s magnum opus was a prank so outlandish that scholars would debate it for hundreds of years. This magic trick was so controversial that it would be immortalized as a symbol of the feud between Cotton Mather and Robert Calef: she levitated to the ceiling in front of witnesses.

In Mather’s telling, from More Wonders of the Invisible World, the miracle is related in a single sentence in a long paragraph full of wonders, not particularly distinguished from any of the other tricks Margaret Rule was performing:

And once her Tormentors pulled her up to the Cieling [sic] of the Chamber, and held her there before a very Numerous Company of Spectators, who found it as much as they could all do to pull her down again.

There it may have stayed, in that lone sentence, if not for the Mather-Calef feud. For as Cotton Mather shifted his attention to the newest bewitched girl, Margaret Rule, Calef came to her chamber to see for himself, and took notes. Calef would end up publishing Mather’s narrative of the Margaret Rule case, Calef’s own notes, their correspondence, Mather’s narrative of the Salem witch trials, Calef’s own narrative of the Salem witch trials, and sundry other letters and arguments related to the scriptural and practical bases of witchcraft. Calef’s book More Wonders of the Invisible World would become the 1700 equivalent of what in modern drama would be an enormous Google document, although he scrupulously presented the materials of each side together, rather than just his own case and supporting evidence.

Cotton Mather was very upset to learn that someone disagreed with him. His letter to Calef in January of 1694 threatened to produce witness statements for a laundry list of claims (in lieu of argument), but in the end, he only produced witness statements for one claim: the levitation of Margaret Rule. “I suppose you expect I should believe it,” Calef replied drily. He published the witness statements with all the rest. Here they are:

I do Testifie that I have seen Margaret Rule in her Afflictions from the Invisible World, lifted up from her Bed, wholly by an Invisible force, a great way towards the top of the Room where she lay; in her being so lifted, she had no Assistance from any use of her own Arms or Hands, or any other part of her Body, not so much as her Heels touching her Bed, or resting on any support whatsoever. And I have seen her thus lifted, when not only a strong Person hath thrown his whole weight a cross her to pull her down; but several other Persons have endeavoured, with all their might, to hinder her from being so raised up, which I suppose that several others will testifie as well as my self, when call’d unto it.

Witness my Hand,
Samuel Aves.

We can also Testifie to the substance of what is above Written, and have several times seen Margaret Rule so lifted up from her Bed, as that she had no use of her own Lims to help her up, but it was the declared apprehension of us, as well as others that saw it, impossible for any hands, but some of the Invisible World to lift her.

Robert Earle.
Copia. John Wilkins.
Dan. Williams.

We whose Names are under-writted do testifie, That one Evening when we were in the Chamber where Margaret Rule then lay, in her late Affliction, we observed her to be, by an Invisible Force, lifted up from the Bed whereon she lay, so as to touch the Garret Floor, while yet neither her Feet, nor any other part of her Body rested either on the Bed, or any other support, but were also by the same force, lifted up from all that was under her, and all this for a considerable while, we judg’d it several Minutes; and it was as much as several of us could do, with all our strength to pull her down. All which happened when there was not only we two in the Chamber, but we suppose ten or a dozen more, whose Names we have forgotten,

Copia. Thomas Thornton.

William Hudson Testifies to the substance of Thorntons
Testimony, to which he also hath set his Hand.

Read over these brief descriptions and see if you can figure out how the trick was done. Of course, it is possible that the fabrication was Mather’s, and that he simply used his social position to pressure witnesses to lie for him. But I think there is a much simpler explanation.

Two of the most prominent Cotton Mather partisans are on record as not being able to figure it out. These two men, William Frederick Poole in 1869 and Chadwick Hansen in 1969, suggest that it’s entirely possible that Margaret Rule magically levitated.

William Frederick Poole was known to be such a shameless Cotton Mather dickrider that it was an academic joke. George Lincoln Burr opens his 1911 paper on New England witchcraft thus:

It is now more than twenty years since I reached the threshold of this theme. Happily it was to learn in time its perils. I was about to read before the American Historical Association a paper on “The Literature of Witchcraft” and my friend Mr. Justin Winsor naturally guessed that it must touch upon New England’s share. “Don’t be afraid,” he encouraged me, “to say just what you please. If Poole pitches into you, I’ll come to your support.”

While Burr assures us that he and Poole remained friends until the latter’s death, he acknowledges Poole’s role in the history of Salem scholarship. In a review of Charles Upham’s book on the history of Salem in 1869, Poole defends Mather from Upham’s perceived attacks (which were in that book mostly relegated to a footnote) thus:

Mr. Upham’s narrative proceeds in the same loose method: “So far was he successful in spreading the delusion, that he prevailed upon six men to testify that they had seen Margaret Rule lifted bodily from her bed and raised by an invisible power so as to touch the garret floor.” This, of course, seems to Mr. Upham very absurd ; but similar instances of elevation are recorded in modern times, and are believed in by those who accept the theory of spiritualism. A bed was lifted in this manner in the house of the Wesleys at Epworth. And Cotton Mather “prevailed upon six men” — Samuel Aves, Robert Earle, John Wilkins, Daniel Williams, Thomas Thornton, and William Hudson — to testify in three depositions to — what ? a fact? Testifying to a fact is a commonplace incident, and divests the statement of all its significance. The inference prepared for the reader is, that Mr. Mather prevailed upon six persons to testify to a falsehood— and all this without a particle of evidence to sustain the charge.

Thus Cotton Mather can count William Frederick Poole among his witnesses to the fact that Margaret Rule could have magically levitated to the ceiling. (I could almost take this as accepting my case that there was a rational but fraudulent explanation, if not for the mention that “a bed was lifted in this manner” at Epworth.) But although almost two hundred years removed from the feats of Mercy Short, Poole lived in a superstitious age in which many people who should have known better accepted Spiritualism and its deceptions as scientific fact.

William Frederick Poole was also the first to publish a summary of the Mercy Short narrative (A Brand Pluck’d Out of the Burning) after its discovery. Burr, about to drop the full Mercy Short narrative to the public in 1914, kindly refers to his old friend Poole’s 1881 summary as “careful,” but he is being more kind than honest. Imagine you have accepted Cotton Mather as your personal savior, and then someone digs out the Mercy Short narrative from behind a drawer in some ancient desk and you are forced to deal with it. Now that we have acquainted ourselves with how foolish Cotton Mather looks from the Mercy Short narrative, we can imagine ourselves in poor Poole’s position. I’m sure we can sympathize, emotionally if not intellectually, with the choice he made—that is, to simply leave out and ignore all the obvious magic tricks that Mercy Short performs. Chadwick Hansen, discussed below, would follow Poole’s lead and perform the same motivated elision.

Chadwick Hansen published his history of the Salem witch trials in 1969 (Witchcraft at Salem). Surely a mid-20th-century English professor wouldn’t believe that a girl could really levitate? But amazingly, Hansen is also willing to provide a witness statement for Cotton Mather. After reproducing all the witness statements I have also reproduced above, Poole says:

These statements might possibly describe an arc de cercle, a violently arched position of the body not uncommon in hysterical fits, which would raise the trunk of the body a considerable distance off the bed. Combined with the power of suggestion to affect bystanders’ senses, this might account for their belief that they had witnessed levitation.

So far so good—it’s a psychological theory, as Hansen leans hard on (mostly 19th century) psychology in his analysis, but at least it’s a theory. Unfortunately, he goes on:

Yet they were not simply bystanders; they were engaging in violent physical activity, trying to bring her body back to the bed. Such activity would, ordinarily, tend to break the power of suggestion. And levitation has been so frequently reported, from so many times and places (from the twentieth-century Indian fakir to the Medieval or Renaissance saint), that one cannot be at all sure there is a satisfactory explanation for it, particularly since so many witnesses insisted that no part of Margaret Rule’s body was touching the bed. In an arc de cercle the head and feet do, of course, touch.

Like Poole before him, Hansen leans toward the belief that Margaret Rule really levitated, for one “cannot be at all sure there is a satisfactory explanation.” (And if you imagined that scholars ceased to make this claim in the 21st century, you would be wrong.) However, now that you’ve had some time to think about it, I’ll reveal what I believe to be a satisfactory explanation for the whole business. Half of the solution can be found in textual clues, and the other half in an ancient item of children’s culture.

The Solution

Hopefully I have established by now that Cotton Mather was almost unfathomably credulous. He was fooled by very primitive magic tricks that do not seem to have convinced the “coffee-house wits.” He demands that others also believe, and frequently insults doubters as being heartless, atheists, or witch-advocates. It would not have taken a David Blaine to convince him of something that he desired with all his heart to believe. The trick could have been very clumsy, but as long as something happened that appeared vaguely like levitation, that would be enough for him to assert it, and to pressure his witnesses to go along with his assertion, even if they had their doubts.

The first piece of the puzzle is that Margaret Rule had accomplices. This can be surmised from many of her magic tricks that would have required confederates, such as the trick of stealing Cotton Mather’s papers and scattering them around Lynn, Massachusetts, for him to find. We even get a clue, from Calef’s account, to the name of one of her potential assistants: Margaret Perd. For example, Margaret Perd volunteered that she smelled brimstone in Margaret Rule’s room, and an unnamed other agreed, while Calef himself did not smell it. From all the available evidence, we can surmise that not only did Margaret Rule have confederates, they were frequently near her bedside.

The second piece is a certain detail from the event which is clearly indicated by the witness statements, which I again reproduce in relevant part:

And once her Tormentors pulled her up to the Cieling [sic] of the Chamber, and held her there before a very Numerous Company of Spectators, who found it as much as they could all do to pull her down again.

I have seen her thus lifted, when not only a strong Person hath thrown his whole weight a cross her to pull her down; but several other Persons have endeavoured, with all their might, to hinder her from being so raised up

it was as much as several of us could do, with all our strength to pull her down. All which happened when there was not only we two in the Chamber, but we suppose ten or a dozen more, whose Names we have forgotten

The critical detail is that in the crowded room, it appeared to the witnesses that many people were trying to pull Margaret Rule down from her levitation. Which is to say, many people were touching her, including, potentially, her confederates.

What if some of that number were not trying to pull her down, but were only pretending to pull her down, and were, in fact, lifting her up? We know from Calef’s account that when witnesses gushed about the strength required to move a bewitched person from their supposedly preternatural postures, they were exaggerating:

…the Attendants said that she was sometimes in a Fit that none could open her Joynts, and that there came an Old Iron-jaw’d Woman and try’d, but could not do it; they likewise said, that her Head could not be moved from the Pillow; I try’d to move her head, and found no more difficulty than another Bodies (and so did others) but was not willing to offend by lifting it up, one being reproved for endeavouring it, they saying angrily you will break her Neck…

So to the extent that some of the people touching Margaret Rule were trying to pull her down rather than trying to lift her, they probably weren’t trying very hard.

The third piece of the puzzle is a beautiful and ancient aspect of children’s culture, and particularly girl children’s culture. This is a “levitation” ritual that is sometimes known in our time as “Light as a Feather, Stiff as a Board.” It was first described in print by Samuel Pepys in his diary in 1665, though it is certainly much older. Given the mysterious ways by which children’s culture maintains itself, it would not be at all surprising if it were in New England in the late 17th century, even in such a notoriously strict culture. Pepys relates that his friend, Mr. Brisband, told him about a ritual Brisband had seen performed by four little girls. They repeat a magic spell, then

putting each one finger only to a boy that lay flat upon his back on the ground, as if he was dead; at the end of the words, they did with their four fingers raise this boy as high as they could reach

To Brisband’s surprise, they could “levitate” a fat cook, too!

The same basic ritual is described by the Opies in their book The Lore and Language of Schoolchildren (1959), again with girl informants (page 309 of my 1987 edition), who participated in the ritual at school in Bath in the 1940s. And the Opies themselves report that they practiced the trick in their own childhoods, giving three variations. Elizabeth Tucker, writing in 2007 in the Children’s Folklore Review, finds that among her Introduction to Folklore students, all the women were familiar with the ritual, but several of the men were not, so the existence of this phenomenon may be news to some readers.

The Opies’ informant says, “It was more like real magic than anything else I have seen.” Indeed, that several people can lift another with only a finger or two each it is a surprising bit of physics that lends itself to the appearance of magic! It is very similar in principle to the phenomenon of crowd surfing, but conducted with more gravitas. If you know any children, you could ask them for more information or a demonstration, or watch one of the performances of the trick on YouTube.

That, I think, is the simplest explanation for how Margaret Rule “levitated”—she did it with a little help from her friends. We know from Calef’s account that her attendants frequently touched her, rubbing and stroking her to make her feel better. It’s easy to imagine a few of her accomplices gathered around tending to her, when suddenly one of them says, “oh look, she’s rising!” Then they all begin to lift her, while crying out for her to come back down, and pretending to pull her down.

I do offer a second possibility, though I think it is less likely. Cotton Mather describes many of the tricks performed by Mercy Short and Margaret Rule (as well as the Salem accusers) in his 1697 biography of Governor William Phips (whom we have previously met when Mercy Short invited him to be pranked). In this text (beginning on page 133, and also functioning as a sort of Greatest Hits of the deceptive magic tricks the girls pulled off) the closest description to the Margaret Rule levitation is this:

or if it were Preternatural to have one’s Hands ty’d close together with a Rope to be plainly seen, and then by unseen Hands presently pull’d up a great way from the Earth before a Croud of People; such Preternatural things were endured by them.

Thornton’s statement says that Margaret was lifted by invisible forces “so as to touch the Garret Floor,” a garret again being an attic, suggesting a space above her where confederates could have concealed themselves with a rope. If Cotton Mather is actually describing the Margaret Rule levitation event with this text, another solution is that she tied the bottom of a rope around herself (perhaps under her covers), and then her confederates in the attic pulled her up with the rope. A tiny bit of support for this theory is that Mather describes Margaret Rule as having been “pulled” up (although all the other witnesses say she was “lifted”), and also uses that verb in this text.

It’s likely, however, that Cotton Mather was describing a different event with this text, or conflating multiple events. While Mather’s use of the words “and then” suggests these events were part of a unified sequence, this may not be the case. Deodat Lawson’s narrative of the Salem witch trials includes a rope trick and a different kind of levitation trick, the latter not being witnessed by a crowd:

Some of the afflicted, as they were striving in their fits in open court, have (by invisible means) had their wrists bound fast together with a real cord, so as it could hardly be taken off without cutting. Some afflicted have been found with their arms tied, and hanged upon an hook, from whence others have been forced to take them down, that they might not expire in that posture.

In other words, the Salem girls hid cords in their sleeves or apron pockets, and secretly and deceptively tied their own hands when no one was looking (a trick that would likely require practice), then claimed that specters did it. Other times they tied each other up to be hanged by a hook like a stereotypical younger sibling or other victim of bullying, again blaming the specters. It’s very likely that Cotton Mather is referring to the Salem events described by Deodat Lawson, and not to Margaret Rule, when speaking of the “preternaturally” tied up girls, especially since neither he nor any other witness mentions a rope in the Margaret Rule case.

It would be odd for Mather to be so impressed with the stunt that he got witnesses to attest to it and then not mention it at all in this “Greatest Hits” selection a few years later, but perhaps he realized in the meantime that it invited especial disbelief. It’s possible that a rope trick is the true explanation for the mysterious event, but I think the much more interesting and clever method inspired by children’s folklore practice is the more likely and textually supported explanation.

What About Salem?

What do the stories of Mercy Short and Margaret Rule, two Boston girls who were not accusers in the Salem witch trials, have to do with the Salem question? First, they matter because, as I stated in the introduction, they bear heavily on the question of Cotton Mather, which is close to the heart of Salem scholars, whether they admit it or not. Second, they are so near in time and culture to the Salem tragedy that they shed light on it, just as do the case of the Goodwin children in 1688 and the Littleton children in 1720, neither of which I have included in this introduction. Burr (1914) puts it,

The importance of the [Mercy Short] narrative lies not only in its contemporaneity with the Salem trials and the side-lights it gives us on that episode and its environment, but yet more in the clearness with which it shows just what its author stood for in the matter. To him the case of Mercy Short was not only identical in kind with those of “the Bewitched people then tormented by Invisible Furies in the County of Essex”: it was itself one of those cases.

Burr’s point is illustrated by Cotton Mather mixing together the events in his Life of Phips, describing Boston and Salem tricks as all part of the same phenomenon.

The same fraudulence and collusion detectable in the behavior of Mercy Short and Margaret Rule is also clear from the records of the Salem accusers. In addition to the rope trick mentioned above, here is the original handwritten testimony of Sarah Nurse regarding the poorly-executed magic trick attempted by Sarah Bibber, accusing Bibber of hiding pins and sticking herself with them to pretend to be afflicted:

I hope to write more about the Salem tragedy, which is why I have labeled this an introduction, but for now I will close with a little recommended reading list, in case the reader is interested in exploring the juicy Salem drama.

One thing that makes Salem the ultimate metaphorical source that other things are compared to, and what makes it rewarding to hobbyist scholars, is that it’s so small. A single person could never hope to be a master of the Cultural Revolution, or the Satanic Panic and related prosecutions of the 1980s and 90s, because in each case there are millions of pages of primary sources spread out over an entire country. But all the primary sources for Salem amount to only a few thousand pages. A solo hobbyist with spare time could learn them in a year, and begin to put them in context by casting a wider net.

The orthography of the late 17th century takes a while to get used to for modern English speakers, but the language is similar enough that it becomes easy to feel the character and personality behind the pen. These people are worth meeting and knowing, even Cotton Mather, and anyone can join the conversation that has been taking place for over three hundred years. The following lists collect some of my recommendations for where to get started in the conversation, with all eras represented.

Cotton Mather Opps, Haters, Alogs, Critics, Detractors, etc.:

Charles Upham, Salem Witchcraft and Cotton Mather: A Reply (1869). The first text I read on the topic, that instantly caused me to become obsessed. Beware!

Robert Calef, More Wonders of the Invisible World (1700). Part Discord leak, part drama callout. Calef’s strategy of printing the exact words of his opponents is laudable. He seems to have truly endeavored to get their best case out of them, so that he could address it, but they never offered much of a case. An interesting read not just for knowledge about Salem, but for a model of how to deal with liars and manipulators. I link a section of the Wikipedia page rather than a single edition because it contextualizes your options for which edition to read (another option is just to download the Burr narratives, linked below, and read it there). The Wikipedia and much of its archived Talk page appear to be written in part by a fellow Cotton Mather alog.

Charles Upham, Salem Witchcraft (1867). It’s a thousand pages long, but it doesn’t feel like it. The first volume is a history of Salem itself, so that when the tragedy in the second volume comes, its weight can be felt. Upham also reprints many useful sources, such as Deodat Lawson’s narrative (a different version from the one published in Burr, below) and various church record entries written by the minister Samuel Parris. (Say what you will about Cotton Mather, but Samuel Parris was a much worse villain in the tragedy.) Upham’s comments about Cotton Mather are mostly relegated to a footnote, which is what triggered Poole to write his review, which in turn triggered Upham to write his 91-page reply close-reading Cotton Mather’s devious prose, a project I hope to expand on at some point.

George Lincoln Burr, Narratives of the Witchcraft Cases, 1648-1706. (1914). Free download in the format of your choice. A valuable compendium of sources, including Cotton Mather’s books about the Goodwin children (before Salem) and Mercy Short (after Salem), Calef’s book including the Margaret Rule narrative, and much more, often with relevant diary entries and context in the notes. He reprints one of the most important documents, Thomas Brattle’s anti-witch-trial letter circulated with only his initials, from 1692 (page 167); both the argument and its author would influence Robert Calef in his work. An excellent choice for your first book on Salem.

George Lincoln Burr, New England’s Place in the History of Witchcraft (1911). A brief but deep synopsis of what made the 17th century witchcraft belief historically unusual, and a measured response to some Cotton Mather sweeping.

Mather-Calef paper on witchcraft. Written in 1694 from Cotton Mather, who is his own biggest opp, to Robert Calef, with the admonition to return it without copying it. This letter formed the basis for an important section of Calef’s text in More Wonders. Lost for centuries and found in 1914, it was published by the Massachusetts Historical Society, together with Calef’s own snarky little notes written in the margin.

Francis Hutchinson, An Historical Essay Concerning Witchcraft (1718). An engaging and frequently funny take on not just the Salem tragedy, but the history of witchcraft persecutions up to that point. Hutchinson was a Church of England minister, not an American, but it’s interesting to see what a person very close in time to the Mather-Calef beef made of it.

Thomas Hutchinson, The history of the province of Massachusets-Bay (1764-1767). Hutchinson, who served as Governor of Massachusetts, had a close family connection with the Mather family, his sister having married Cotton Mather’s son. He is polite about avoiding having Cotton Mather’s literal name in his mouth, but still roasts him in clearly identifiable ways (see Upham 1869, linked above, page 69, for examples). Hutchinson was born (in 1711) almost a generation after the events of Salem, but close enough in time for us to give deference to his impressions; also, he was working with a more complete set of records than later writers.

Cotton Mather Glazers, Sweepers, Dickriders, Defenders, Supporters etc.:

William Frederick Poole, Cotton Mather and Salem Witchcraft. 1869. The book review that Upham’s 91-page reply was written in reply to, see above. This is where he suggests that Margaret Rule really magically levitated. Poole also wrote a reply to Upham’s reply which I felt to be a personal attack:

William Frederick Poole, Witchcraft in Boston. Chapter IV of The Memorial History of Boston (1881). Begins on page 131. The narrative of Mercy Short, as told by a Cotton Mather fan—which is to say, with all the fun stuff edited out.

Wendell Barrett, Were the Salem Witches Guiltless? (The talk was presented in 1892, though the book was published in 1893. Begins on page 65.). Another example of an author who feels compelled to defend Cotton Mather by arguing that the Salem witchcraft was real, and that at least some of the executed deserved to die, like Poole drawing an analogy to 19th century Spiritualist practices.

George Lyman Kittredge, Notes on Witchcraft (1907). Excusing Cotton Mather from responsibility by arguing that the witchcraft belief was ubiquitous through all time, that everyone always believed in it, and that Cotton Mather was not particularly a moral entrepreneur who attempted to “get up” witch persecutions. This is the text that Burr (1911) is replying to.

Thomas J. Holmes, The Surreptitious Printing of One of Cotton Mather’s Manuscripts (1925). Useful as a brief synopsis of the sweeper case. Especially amusing for the insults and invective thrown at Robert Calef. Holmes, like Poole, was a librarian, but Holmes was more specifically the librarian of William G. Mather’s library, William G. Mather being an industrialist who was a direct descendant of Cotton Mather’s uncle, Timothy Mather (brother of Increase).

Chadwick Hansen, Witchcraft at Salem (1969). Easily the worst book about Salem that I have ever read, and one of the worst books on any subject I’ve ever forced myself to read, and I hate-read Francesca Gino’s book. Hansen is a Mather dickrider of a caliber not seen since Poole, and argues that not only was witchcraft real, the witches deserved to be hanged, and hanging was a lenient sentence. He carefully avoids mentioning any evidence that does not support his outlandish theory. I read some of his other work, which was much more interesting and reasonable. In his other work, he made one tiny reference to a specific trickster cultural practice, that is, speaking in such a way that outsider listeners are not able to understand what one is up to, so I cling to a fantasy that his book was a joke that nobody got, and when it ended up being his most popular work, he couldn’t exactly reveal the truth. It’s full of credulous claims of sketchy psychological and folkloric phenomena that had me fact checking him like he was a bad science twitter account. Terrible book, but a lot of fun for a banana.

Fence sitters:

Bernard Rosenthal, Records of the Salem Witch-Hunt (2009). A compendium of all the trial documents ever collected from Salem and the nearby towns that the witch trials spread to, all in one tidy volume. (You can get earlier versions for free, such as W. Elliot Woodward’s two-volume set Records of Salem Witchcraft printed in 1964, or the University of Virginia online repository which includes images of the original handwritten records, but Rosenthal’s edition is admirably complete and handy to have.) The introductory essay rushes to assert Cotton Mather’s innocence and good conduct (leading off with “Mather was instrumental in containing the episode [of the Goodwin children], which claimed no victims other than Glover, convicted of witchcraft and executed.” is certainly not a choice I would have made after reading Mather’s entire account in Memorable Providences). Rosenthal also praises Chadwick Hansen’s work, which I also find inexplicable, but he at least acknowledges the fraud committed by the accusers. Placing him in this category is more of a statement about how extreme the Cotton Mather sweepers get than about his apparent allegiance.

Emerson Baker, A Storm of Witchcraft (2015). Possibly the only recent history that didn’t trigger me to have a tantrum like Mercy Short in church. Avoids taking a side without particularly sweeping. Extra points for pointing out that psychological theories for the accusers’ behavior are controversial, rather than pretending, as some do, that a psychological theory is the consensus.

Thank you to my friend Curtis Yarvin, who introduced me to the story of Robert Calef, and who was brave during the scary time.

AI robot voice version! not made by me

Introductory summary: The current scientific consensus is that the placebo effect is a real healing effect operating through belief and suggestion. The evidence does not support this. In clinical trials of treatments, outcomes in placebo and no-treatment arms are similar, distinguishable only in tiny differences on self-report measures. Placebo-focused researchers using paradigms designed to exploit demand characteristics (politeness, roleplaying, etc.) produce implausibly large effects, in many cases larger than the effect of fentanyl or morphine, but these studies measure response bias on self-report outcomes (at best). There is no evidence that placebos have effects on objective outcomes like wound healing. Three sources of evidence purport to show that the placebo effect is a real, objective phenomenon: brain imaging studies, the alleged involvement of the endogenous opioid system or dopaminergic system, and animal models. But the brain imaging studies do not demonstrate an objective effect, but are rather another way of measuring “response bias,” as subjects are capable of changing these measures voluntarily. Studies that claim to demonstrate the involvement of the endogenous opioid system suffer from replicability issues, with most positive results coming from a single laboratory genealogy; other laboratories produce conflicting results. Animal models also suffer from replicability issues, such that the highest-quality research is least likely to produce a placebo effect in animals. Even research designs that do produce a conditioned “placebo effect” in animals cast doubt on the involvement of the endogenous opioid system. In the era of open science, there has been no large-scale, multi-center, preregistered attempt to address the placebo effect in animal models or the involvement of the endogenous opioid system. The one adequately powered preregistered attempt for the dopaminergic system in humans produced no effect. Although placebo and “mind-cure” beliefs are widespread, the most parsimonious interpretation of the evidence is that the “placebo effect” is not a real healing effect, but a product of response bias and questionable research practices. The true power of the placebo is as a blind.

MB: I’ve detected a note of skepticism from you regarding the effects of placebos having actual physical effects on the body, but my rough understanding of it is that it’s not a priori totally ridiculous. The mind and the body (if we’re gonna do Cartesian dualism) are sort of intimately related; they both have effects on the other. And if you believe something is true, then it’s gonna have these psychological effects which are gonna translate into physical effects at some point, right?

CK: Oh no – yeah I have no skepticism about the placebo effect existing, or like you say, what people expect about things, you know where else is it gonna show up? Even the fact that the people subjective self-reporting is different, it wouldn’t be a surprise for me to find that you could find correlates between activity that represents that. So it’s not a huge reach.

(Matthew Browne and Christopher Kavanagh, Decoding the Gurus podcast, November 9, 2023, at about 2:29:25, speculative punctuation mine. Podcast beginning at 1:46:60 assigned in Daniel Lakens’ metascience class.)

The context of the above excerpt is our two scholars very politely and decorously tearing apart some sketchy placebo studies. It is obvious from the podcast that they are both highly intelligent and highly aware of the crisis in the sciences. They demonstrate not only the capability to identify specific flaws in studies, but also a fine olfactory sense for identifying questionable claims by smell. I open my case against placebos with an excerpt from their podcast not because I think they are fools, but precisely the opposite: it seems to be the consensus among intelligent, aware people that the placebo effect is real, and my own claim that the placebo effect is not real is in conflict with this consensus.

It often strikes me as odd, as a rare placebo effect denier, that even while criticizing studies that find, for example, that simply being told that one’s job is good exercise causes people to lose weight, critics take care to say they do not question that the placebo effect exists. The placebo effect as a healing effect is taken for granted, partly because it is not widely understood how shaky the evidentiary foundation for the healing placebo effect is.

Here is a list of sub-beliefs, which I hope to show are misconceptions, that support the belief in the healing placebo effect, some of which can be inferred from the above quotation:

• Randomized placebo-controlled trials use a “placebo arm” because the placebo effect is known to be powerful.
• The placebo effect is a real healing effect, and not just subjects being polite on self-report instruments, regression to the mean, selection on extreme values at the beginning of a study, or questionable research practices.
• The placebo effect is a healing effect that is large enough to be noticeable and clinically relevant.
• The placebo acts not only on subjective states, but also on objective, physical outcomes that are measurable by e.g. laboratory tests.
• We know the placebo effect is objective and not a function of response bias because it causes measurable changes in The Brain, for instance measured in the EEG sustained late positive potential.
• The placebo effect need not be deceptive; it may be evoked with “open-label placebos.”
• We know the placebo effect is an objective phenomenon because it involves endogenous opioids, demonstrated by high-quality and well-replicated research in which the placebo effect is abolished with a hidden injection of an opioid antagonist.
• We know the placebo effect is real because we can induce it in animal models with a conditioning paradigm.
• Expensive placebos are more effective; discount placebos are less effective. Properties of the placebo including the color of the tablet alter the subsequent placebo effect.

Some people are convinced that the placebo effect must exist, because they notice when children receive a mild injury, kissing the boo-boo or perhaps applying a band-aid seems to soothe any upset. But I don’t think this is a healing placebo effect at all. This is a rational process of people who are new to the world encountering a disturbing situation, and going to people they trust with more experience to find out how freaked out they should be. Deciding how upset to be based on contextual information is different from healing. (Also band-aids are pretty effective at protecting wounds from being touched, preventing more pain.) Receiving a placebo is a communication that conveys, for example, that no further treatment is forthcoming. It can mean anything between “it’s okay” and “shut up.” Recipients of placebos take this into consideration and may alter their communication in accordance with this, but they do not actually heal.

In faith-healing events like revival meetings, believers may throw away their crutches, eyeglasses, or hearing aids, and this voluntary behavior serves as a communication of faith. However, this behavior is not evidence that they no longer need these helping devices, as they may find themselves sheepishly buying replacements when the strong emotion of the meeting has worn off. It’s important not to confuse communication strategies with healing.

“To amuſe the mind”

It is well known that the placebo has something to do with pleasing, translating as “I shall please,” but it is a bit unclear who is to be pleased. I have sometimes heard that medical students are taught that placebos are given to patients so that they will get better to please their doctors! Dr. J. C. Lettsom, recorded by William Gaitskill in the Memoirs of the Medical Society of London in 1795, gives his patient placebos “to amuſe the mind,” implying the patient is the one to be pleased. I suspect that both senses coexisted throughout the past few hundred years. Some doctors (of various levels of respectability) sell placebo treatments because people like them and want to buy them; others expect placebos to actually work to some degree.

In a lecture given in 1953, Sir John Gaddum smears the two meanings together – a placebo may both please the patient and act to cure him through psychological suggestion – but he offers a further distinction which will be relevant in the next section:

Such tablets are sometimes called placebos, but it is better to call them dummies. According to the Shorter Oxford Dictionary the word placebo has been used since 1811 to mean a medicine given more to please than to benefit the patient. Dummy tablets are not particularly noted for the pleasure which they give to their recipients. One meaning of the word dummy is “a counterfeit object”. This seems to me the right word to describe a form of treatment which is intended to have no effect and I follow those who use it. A placebo is something which is intended to act through a psychological mechanism. It is an aid to therapeutic suggestion, but the effect which it produces may be either psychological or physical.

As we shall see, it might have prevented a great deal of confusion to preserve the distinction Gaddum makes between placebos and dummies:

Dummy tablets may, of course, act as placebos, but, if they do, they lose some of their value as dummy tablets. They have two real functions, one of which is to distinguish pharmacological effects from the effects of suggestion, and the other is to obtain an unbiased assessment of the result of the experiment.

Gaddum is concerned with the usefulness of placebo (dummy) controls in revealing the efficacy (or lack thereof) of drug treatments. He gives as an example the drug thonzylamine, of which “extravagant claims were made” of its ability to prevent and treat the common cold. But when compared with placebo control, the typical result was this:

That is, the placebo control (dummy) revealed that the apparent efficacy of the drug was down to factors unrelated to the drug. Pre-post comparisons without control could not have revealed this. But what this kind of experiment cannot reveal is whether the improvement in the placebo group was down the “psychological” effects of suggestion (mind cure), or to regression to the mean, natural history, or even politeness of the subjects (depending on how “cure” was assessed).

The Placebo Effect in Placebo-Controlled Trials

One of the most common misconceptions about placebos it that placebos are used in randomized controlled trials because the placebo effect is known to be a powerful healing effect. This misconception is addressed by Blease et al. in a paper from 2020, Open-label placebo clinical trials: is it the rationale, the interaction or the pill? They distinguish, on the one hand, the placebo response in placebo-controlled trials, in which patients in the placebo arm often actually do “get better” in a pre-post analysis (a dummy, in Gaddum’s terms), from, on the other hand, the placebo effect, which is the purported healing effect of the placebo.

They grant, as I do, that “there is a scientific consensus that placebo effects constitute genuine psychobiological events that engage perceptual and cognitive processes to produce therapeutic benefits among patients for a range of self-reported conditions and symptoms.” (To be clear, I think this is wrong, but I acknowledge that it is the current scientific consensus, unfortunately.) But the use of placebos as a control in clinical trials is agnostic to these purported healing effects. The placebo in a placebo-controlled trial exists, for one thing, to ensure blinding, so that any measured effects can be attributed to the treatment, rather than to the hopes and beliefs of researchers. Subjects in the placebo arm experience the same passage of time as the treatment arm, so that noise and any regression to the mean can be subtracted out and not mistaken for a healing effect of the treatment. Subjects given placebo are in the same position as subjects given a treatment in terms of being motivated to respond politely to surveys, such that theoretically they should have a similar “response bias” as treatment subjects, if the blind is intact. (Though note that we should expect them to respond politely even if they are given an open-label placebo with a rationale for why it is expected to work.) If “eligibility creep” is a factor, in which the measurements of subjects at baseline are exaggerated to qualify more subjects for the trial, it should be the same for the placebo and the treatment group (this is why “placebo effects” sometimes occur on objective outcomes in randomized controlled trials, especially those without a no-treatment arm for comparison, even though placebos only affect self-report measures).

For all these reasons, any pre-post improvement in a placebo arm (endpoint minus baseline) is not necessarily attributable to the placebo itself. To measure the effect of placebo, one must somehow design a trial comparing placebo to no treatment – but even then, subjects in the no-treatment arm are not given the same motivation to answer politely to surveys. (This is among many reasons why some have questioned whether the waiting list is a “nocebo” control condition, that is, worse than nothing, which makes sense – subjects assigned to a waiting list are invited to portray themselves as still in need of help to potentially qualify for the trial, whereas subjects given at least a placebo no longer need to qualify for the trial. Also, researchers doing the rating may encounter subjects coming to the laboratory for treatment or placebo, but may never encounter subjects assigned to the waiting list, threatening the blind.)

It’s impossible to achieve a blind when comparing placebo to no treatment, since the placebo itself is the main method for blinding in the first place. This is the promise behind the paradigm of hidden injection of opioid antagonists, which we shall address in a later section.

The Powerful Placebo

One of the most influential documents in favor of the scientific view of the healing placebo effect is Henry Beecher’s 1955 paper, The Powerful Placebo. Perhaps we can blame Dr. Beecher for the loss of Dr. Gaddum’s useful distinction between placebos and dummies, for Beecher, quoting Gaddum as I have above, specifically argues that they are the same thing:

Both “dummies” and placebos are the same pharmacologically inert substances; i. e., lactose, saline solution, starch. Since they appear to be differentiable chiefly in the reasons for which they are given and only at times distinguishable in terms of their effects, it seems simpler to use the one term, placebo, whose two principal functions are well stated in Professor Gaddum’s last sentence quoted above. Finally, I do not understand how a dummy tablet could be prevented from having a psychological effect that, if pleasing, would make it a placebo. One term seems to fill the bill. If it falls a bit short of precision, perhaps the language will have to grow a little to include the new use.

Indeed, that is exactly what happened!

Writing in 1955, Dr. Beecher performs what would now be regarded as a rather crude meta-analysis of his own work and that of others:

Fifteen illustrative studies have been chosen at random (doubtless many more could have been included) and are shown in table 2. These are not a selected group: all studies examined that presented adequate data have been included. Thus in 15 studies (7 of our own, 8 of others) involving 1,082 patients, placebos are found to have an average significant effectiveness of 35.2±2.2%, a degree not widely recognized. The great power of placebos provides one of the strongest supports for the view that drugs that are capable of altering subjective responses and symptoms and do so to an important degree through their effect on the reaction component of suffering.

That is, rather than presenting a modern effect size, he concludes that the pain of 35.2 plus or minus 2.2% of subjects were “satisfactorily relieved by placebo,” which he later defines as follows:

For example, in our pain work satisfactory relief is defined as “50 per cent or more relief of pain”at two checked intervals, 45 and 90 minutes after administration of the agent. (This is a reproducible judgment patients find easy to make.) Each author has been explicit, and some have required even greater success than indicated above. For example, Gay and Carliner (1949) required, for a positive effect, complete relief of seasickness within 30 minutes of administration of the placebo.

That is a powerful placebo! While Dr. Beecher focuses on “altering subjective responses” and the “reaction component of suffering,” he also claims that placebos are effective on objective, “physiological” criteria. We will return to this claim in a later section.

For now, we will introduce two of the few placebo skeptics I have encountered, Gunver Kienle and Helmut Kiene, who in 1997 published The Powerful Placebo Effect: Fact or Fiction? in response to Beecher’s claims. Among many other criticisms, some of which we will return to, they take issue with Beecher’s reporting of his fifteen trials:

Beecher misquoted 10 of the 15 trials listed in “The Powerful Placebo.” He sometimes inflated the percentage or the number of patients, or he cited as a percentage of patients what in the original publications is referred to as something completely different, such as the number of pills given, the percentage of days treated, the amount of gas applied in an experimental setting, or the frequency of coughs after irritating a patient. The main effects of these errors were false inflations of the alleged placebo effect. A multitude of misquotations can also be found in other placebo literature. (Citations omitted.)

Kienle and Kiene consider misquotation to be “a particular problem of placebo literature,” but in my experience it is a problem of all literatures ever.

Run In, Wash Out

One important aspect of Dr. Beecher’s work is in his conception of a “placebo responder.” That is, some patients respond to placebo, and some don’t, depending on “attitudes, habits, educational background, and personality structure” (but not intelligence!) of the subjects.

Dr. Beecher seems to anticipate the method of “placebo washout” that was later commonly used in antidepressant trials, that is, to deceptively provide subjects with a placebo said to be an effective treatment for a week or two, and then exclude any subjects who report getting better. He says, “as a consequence of the use of placebos, those who react to them in a positive way can be screened out to advantage under some circumstances and the focus sharpened on drug effects.”

This is an important prediction that has now been falsified, at least for a well-studied class of drugs purported to be affected by placebo response, antidepressants. Although there has long been criticism of the practice of placebo washout (also called placebo run-in) as a questionable research practice in antidepressant research (e.g., “we think that the practice of excluding patients during the washout procedure should be suspended due to the potential for distorting results in some studies,” Antonuccio et al. 1999), the actual effect of the procedure turned out to be nil.

As early as 1995, Greenberg et al. (confirming, they report, the results of two earlier studies) found that measures of depression were not affected either in placebo or drug groups from the use of a washout design:

Our results are entirely consistent with the few attempts in the literature to assess the value of the placebo-washout technique. We found no significant difference between washout and nonwashout studies in the percentage of reduction in ratings on depression for subjects in the placebo groups…. There was also no difference in the effectiveness of antidepressant drugs in the two types of studies…. Our analyses showed equivalent percentages of dropouts in the two types of studies for both the patients assigned to the placebo groups….

In 2021, Scott et al. examined a much larger sample of trials and found that while washout (placebo run-in) designs produced slightly lower placebo effects and slightly lower drug effects, the drug-placebo difference was indistinguishable between methodologies:

Studies using PRI periods reported a smaller placebo response (g = 1.05 [95% CI, 0.98-1.11]; I² = 82%) than studies that did not use a PRI period (g = 1.15 [95% CI, 1.09-1.21]; I² = 81%; P = .02). Subgroup analysis showed a larger drug response size among studies that did not use a PRI period (g = 1.55 [95% CI, 1.49-1.61]; I² = 85%) than those that did use a PRI period (g = 1.42 [95% CI, 1.36-1.48]; I² = 81%; P = .001). The drug-placebo difference did not differ by use of [placebo run-in] periods (g = 0.33 [95% CI, 0.29-0.38]; I² = 47% for use of a [placebo run-in] period vs g = 0.34 [95% CI, 0.30-0.38]; I² = 54% for no use of [placebo run-in] periods; P = .92). The likelihood of response to drug vs placebo also did not differ between studies that used a [placebo run-in] period (odds ratio, 1.89 [95% CI, 1.76-2.03]) and those that did not use a [placebo run-in] period (odds ratio, 1.77 [95% CI, 1.65-1.89]; P = .18).

Excluding subjects who are particularly willing to play along slightly reduces pre-post differences for both placebo and treatment, but doesn’t affect the drug-placebo difference.

On the other hand, if a researcher wants to produce large placebo effects, selecting only “placebo responders” exploits response bias by excluding subjects not willing to play along. Many research designs described below use this method, particularly in the section on endogenous opioids.

The question of whether placebo responders exist, as a consistent natural kind, seems to be an open one. In a small (n=71) study, Whalley et al. found that while responses to a placebo cream of the same name in different trials were somewhat correlated, there was no significant correlation in responses to placebos given different names. Given my beliefs about the placebo effect, that it is primarily a function of politeness and roleplaying in self-report measures on the part of the subjects, it would not be surprising if some subjects are more polite and better sports than others, but I cannot find much evidence of a consistent placebo responder in the literature. This does not stop many authors from conducting sketchy subgroup analyses of only “placebo responders” to find a placebo effect.

What Outcome Measures Are “Objective”?

I have claimed that the “placebo effect” is exclusively a phenomenon of self-report or subjective measures, and never objective measures. However, this distinction needs some clarification, as there are many different uses of “objective” in the literature. I will review some of these definitions here, as they will be relevant in the sections to come.

One possible meaning is “anything other than a self-report measure,” and this produces three distinct kinds of problems. First, this may include observer-reported outcomes that constitute a summary of a patient interview, such as the Hamilton Depression Rating Scale. Rather than producing an objective measure, like a laboratory test of a blood sample, this combines the subjective impression of two parties. A patient in a trial who has received a placebo may feel encouraged to role-play as if he has improved. But a researcher rating improvement may also interpret ambiguous information as favoring improvement. Indeed, researchers rating subjects produce larger pre-post effects in both placebo and treatment arms of depression trials, as discussed in a later section. Few such trials include a no-treatment arm to pinpoint a “placebo effect,” but the few that do indicate that it is modest, if it exists at all. In short, researchers may exaggerate on essentially subjective and vague criteria even more than subjects role-play as having improved. When the outcome does not allow for exaggeration or roleplaying, such as laboratory tests or wound healing, there is no “placebo effect.”

A second problem is that this definition seems to exclude brain imaging, as an EEG or fMRI result is not, strictly speaking, a self-report measure. However, available evidence suggests that just as verbal or written responses are produced voluntarily and may be changed depending on politeness or roleplaying, many brain imaging outcomes are also under voluntary control. Like the rate of breathing or facial expression or amount of effort exerted, subjects have control over these outcomes, even if they are measured in a manner that doesn’t superficially look like self-report.

The third problem is specific to pain measurement. A rating on a pain scale is clearly a self-report measure – for example, “How bad is your pain, on a scale of 1 to 100?” clearly asks for a self-report that may change based on how politely the subject is playing along. Other pain measures are sometimes termed “objective” if they are not strictly measured by self-report on a pain scale. Examples include the temperature in a heat pain protocol at which the subject reports unbearable pain, or the length of time a subject is able to endure induced pain. However, even though not a pain rating, these “objective” measures are under the control of the subject and may still be a product of roleplaying. However, I do not mean to say that large placebo effects on this kind of pain measure are only produced by roleplaying, as they may also be produced by questionable research practices. For example, no laboratory in the open-label placebo meta-analysis discussed in a later section, Spille et al. (2023), produced a placebo effect using such an “objective” pain measure, but certain laboratories claim to produce enormous effects of this kind, suggesting some difference in methodology that might be regarded as questionable.

The distinction between outcomes that are under voluntary control (and therefore subject to response bias) and truly objective outcomes is a bit fuzzy. Are exercise performance outcomes, for example, objective? Exercise science is still, to put it politely, in the early stages of addressing the replication crisis (with, as Büttner et al. 2020 report, 82.2% of studies reporting that the primary hypothesis of the study was supported, despite, as Abt et al. 2020 report, a median sample size of 19 in a random sample of papers in the Journal of Sports Sciences). However, it is still possible, in my model, that at least some placebo effects on exercise outcomes (time to run a certain distance, for example) are not products of sketchiness and fraud. Effort in exercise is under voluntary control, and certainly influences exercise outcomes as measured in trials. Differential effort, rather than fraud, may even explain placebo effects in studies in which the authors give the subjects a survey about their level of effort and find no difference, as there is no reason to suppose that a survey instrument is a reliable way to retrospectively measure effort during athletic performance.

To me, an objective outcome is not under the voluntary control of the subject and unlikely to be manipulated by researchers. Some examples of this kind of objective outcome would be wound healing, laboratory blood or urine tests, or pregnancy. Cognitive tests may also be objective measures as long as effort is constant, and indeed cognitive tests do not seem to show a “placebo effect” unless they are produced by the laboratories of known frauds.

The issue of the objectivity of outcomes is also relevant to a distinction our placebo skeptics Keinle and Keine (1997) make between the placebo effect and the concept of “psychosomatic” effects. They say, just before noting the “uncritical reporting of anecdotes” in the pro-placebo literature:

There is a class of anecdotal reports in the placebo literature, which have nothing to do with placebos, because no placebos were given at all.

The purpose of these anecdotes is to demonstrate the possible power of “nonspecific” causes. Beecher himself reported adventurous episodes from the voodoo culture, when supposedly dying people recovered immediately, or when magic rituals brought about the death of apparently healthy people.

Another classic example is an anecdote in Stewart Wolfe’s well known “The Pharmacology of Placebos:” A woman with a gastric ulcer could not respond with gastric acid production during provocative tests with even the most powerful secretory drugs. Yet, immediate acid secretion occurred when she was asked about her husband who, as she had just recently discovered, had been sexually abusing her 12-year-old daughter. Wolf used this story to demonstrate the possible range of placebo effectiveness. However, this is misleading. This was an example of a psychosomatic effect, not the effect of placebo application. The example does not show that the mere ritual of giving a pill can be equated with the effect of discovering the sexual abuse of one’s daughter by one’s husband.

It is worth making a distinction between placebo effects and psychosomatic effects, but it is also worth making a distinction between the measurable objective outcomes of strong emotion (e.g. heart rate increase, crying) and the possible effects of emotion or “mindset” on disease outcomes. For the former, it would be strange for emotions to evolve at all if they had no effects. For the latter, the picture is murkier. For example, for peptic ulcer, a condition widely believed to have psychosomatic causes, hopelessly confounded observational studies often show a link between some measures of stress and some measures of ulcer. But it does not follow that peptic ulcer responds to placebo. Within clinical trials testing a treatment for peptic ulcer, de Craen et al. (1999) found a small difference between trials that had subjects take four placebos, as opposed to two placebos, per day, but even their small result was not robust to various sensitivity checks. (Also, one could imagine that e.g. drinking two extra glasses water could have some small effect on a gastric outcome.) Claims that a “cancer-prone personality” caused cancer and could be treated with talk therapy turned out to be based on fraud, as we will see in a later section. It is worth keeping placebo claims separate from vague claims of psychosomatic effects, but it is also worth treating claims of psychosomatic effects on disease with skepticism.

Powerless Placebo?

Massive pre-post effect sizes in the placebo arms of placebo-controlled trials can be confused with a large placebo effect (for example, Hedge’s g greater than one for both placebo groups in antidepressant trials analyzed in Scott et al. cited above!). However, any effect of placebo itself must be distinguished from effects from the passage of time (especially in conditions of an episodic nature), and related issues such as inflation of initial scores. A “no treatment” or natural history control group can accomplish much of this. Although the inclusion of a no-treatment group won’t match effects of politeness and roleplaying (“response bias,” “Hawthorne effect,” “demand characteristics,” etc.), it goes a long way toward establishing the true effect of placebo.

Hróbjartsson and Gøtzsche (2001) performed a meta-analysis of studies comparing placebo arms to no-treatment arms (updated in 2004, and again in 2010). Their conclusions were surprising to placebo believers: “We found no evidence of a generally large effect of placebo interventions. A possible small effect on patient-reported continuous outcomes, especially pain, could not be clearly distinguished from bias. (2004)”

There are some interesting aspects of this analysis. Consistent with the politeness hypothesis, the authors found a significant (but surprisingly small) effect of placebo versus no treatment on self-reported outcomes. They found no such effect for observer-reported outcomes. They only found a placebo effect on continuous outcomes, and a perfect null for binary outcomes. They generously conclude, “We have no good explanation for the difference between effects of placebo when measured on a binary and on a continuous scale, but continuous scales could be more sensitive to small effects or biases. (2004)” The binary outcomes produce one of the loveliest funnel plots I’ve ever seen, almost perfectly symmetrical with a peak at the null value (2004):

Interestingly, there is a small but statistically significant effect of placebo on the subgroup of continuous outcomes measured by laboratory tests (2004) – in the wrong direction. That is, in laboratory tests with continuous outcomes, subjects given placebo did slightly but statistically significantly worse than subjects given no treatment. This is almost certainly noise, but it’s important to take note of absurd results from noise, as if it had gone in the other direction through chance, it might have been taken as evidence of placebo efficacy. To take another example, So et al. (2009) studied the effects of acupuncture and sham acupuncture on various objective outcome measures during IVF treatment. They found that the sham acupuncture group (the placebo group) had a statistically significantly higher overall pregnancy rate, one of three measures reported, with a p value of .038. I have described their conclusion as a two-sentence horror story:

Placebo acupuncture was associated with a significantly higher overall pregnancy rate when compared with real acupuncture. Placebo acupuncture may not be inert.

Of course, it’s just noise, as you might suspect from the p value. Coyle et al. (2020), meta-analyzing data from eight trials and almost ten times as many subjects as So et al., found no effect of acupuncture or placebo compared to usual care on any outcome measure after IVF. I have a general maxim: if there is a statistically significant effect of placebo on an objective outcome, it is either noise, fraud, questionable research practices, or a mischaracterization of a subjective outcome as objective.

Interestingly, although Hróbjartsson and Gøtzsche found a placebo effect for self-reported but not for observer-reported outcomes, in the case of antidepressant trials versus placebo, “placebo effects” seem to be larger when a researcher is doing the rating. In trials of depression treatments, both the placebo effect (Rief et al., 2009) and the treatment effect (Cuijpers et al., 2010) are larger for clinician-rated effects compared to self-reported effects. One interpretation of this is that the “placebo effect” in these trials is not so much from patients being polite and exaggerating their benefit, but from researchers exaggerating the change, either innocently or for the purpose of producing a larger apparent effect. (More on the placebo in depression in the appendix.) Interestingly, a 2010 meta-analysis reported that this was also the case for Irritable Bowel Syndrome: higher placebo response rates for physician-reported than subject-reported outcomes. The negative result for outcomes measured by laboratory tests suggest this is exaggeration, rather than a genuine objective improvement.

But the main upshot of Hróbjartsson and Gøtzsche (2004) is how small the placebo effect is. For example, “pain” was the subgroup that performed the best (by its nature a self-report subgroup prone to bias), but the effect was estimated at only 6 points on a 100-point scale, too small to be clinically relevant. To make matters worse, Kamper et al. (2008) conducted an updated meta-analysis of the pain trials in search of trial characteristics that might produce a large, clinically relevant placebo effect (“trial-design, patient-type, or placebo-type”), and found none, but also found the placebo effect on pain to be a mere 3.2 points on a 100-point scale. As they put it, “Our analysis confirms the conclusions of Hróbjartsson and Gøtzsche that, at least in the context of clinical trials, placebo interventions appear to have little effect on pain.”

The “at least in the context of clinical trials” part is important, because researchers in ordinary clinical trials have little incentive to put their thumbs on the scale in favor of a placebo effect. Researchers specifically designing trials to find a large placebo effect are more likely to “find” one.

Here is Kamper et al.’s plot for the pain studies, which also tells a story:

Of course, as all these authors are aware, the small placebo effect on self-reported pain does not answer the question of whether it is real pain relief or just subjects being polite. But assuming for a moment that it is 100% real pain relief, how much pain relief is enough to matter? This is a tricky question, but to take a stab at it, Olsen et al. (2017) (which includes our friend Asbjørn Hróbjartsson as an author) analyzed 37 studies that sought to investigate the magnitude of the clinically important difference in pain, based on mapping numerical scores to patient report of feeling better:

A typical eligible study would ask patients to score their pain intensity, e.g. using a VAS, at baseline and follow-up. At follow-up, patients were also asked to categorise their change in pain intensity using response options such as ‘no change’, ‘a little better’/‘somewhat better’, and ‘a lot better’/‘much better’. The [minimum clinically important difference] was then determined from the change in scores on the pain scale among patients having categorised their change as ‘a little better’ (or a similar expression indicating a minimum clinically important improvement).

Olsen et al. found that the studies varied widely in their conclusions, ranging from 8 points on a 100-point scale to 40 points (points may also be expressed as millimeters on a visual scale). No study found a clinically relevant difference as low as 6, much less 3, points on a 100-point scale. Far from Dr. Beecher’s claims of 50% reduction of pain in over a third of patients, in the typical case, placebos do not seem to make people report feeling even “a little better” in clinical trials compared to no treatment.

In a final update as I am writing, Hohenschurz-Schmidt et al. (2024) confirmed that in three-armed trials (clinical trials of a treatment with a placebo and a no-treatment arm), short-term placebo effects on pain are small, and medium-term and long-term placebo effects are nonexistent.

A large “placebo effect” is only demonstrated when researchers are determined to find it, as we will see in the next section.

Placebo-Controlled Trials versus Placebo Trials

Vase et al. (2002), in A comparison of placebo effects in clinical analgesic trials versus studies of placebo analgesia, found that studies that were trying to find a placebo effect indeed got a larger placebo effects (mean .95) than clinical trials that happened to have a placebo and a no-treatment arm (mean .15). Of the fourteen placebo-focused studies they review, eight were from the laboratories of Levine or Benedetti, discussed in a later section.

Hróbjartsson and Gøtzsche raised some methodological issues both with the analysis itself and the studies included in a letter, Unreliable analysis of placebo analgesia in trials of placebo pain mechanisms (2003). The funniest one is an issue that still plagues low-quality science to this day: “results from the included studies were summarised as simple unweighted averages.” They also question study quality: “Thirteen out of 14 mechanism studies did not report the method of concealment. The only study of pain mechanism with a clearly concealed allocation of patients reported a minor effect of placebo (Roelofs et al., 2000).” (Roelofs et al., 2000, is reviewed in the appendix on opioid antagonists.)

Price et al. (2003) (the authors of Vase et al. 2002) respond, in Reliable differences in placebo effects between clinical analgesic trials and studies of placebo analgesia mechanisms (which differences I do not think are the flex these authors think they are). They say,

Admittedly, we did not use a method of weighting studies in order to keep the analyses simple. However, a weighted estimate of d does not change our conclusions, rather it strengthens them.

They produce an even bigger result with a weighted analysis! They also say:

It is true that we did not have concealed allocation of patients as an
explicit inclusion criterion. However, all studies of placebo mechanisms in our analysis included random allocation of subjects, and nothing indicates that the randomization was performed in a way that was clearly not concealed. So in that aspect, inclusion of studies in our meta-analysis was no different from inclusion of studies in Hrobjartsson and Gøtzsche’s meta-analysis (2001). Roelofs et al. (2000) described the randomization procedure in great detail and they reported a minor placebo analgesia effect. However, there is no evidence that the randomization procedure was related to the magnitude of their reported placebo effect.

While Hrobjartsson and Gøtzsche had concerns with the methodology of the meta-analysis, I think the real reason for the difference in effect sizes is the difference in methodologies between the clinical trials of treatments and the placebo-focused studies. It’s about study quality and researcher motivation. Proponents of placebo effects claim that designs in clinical trials aren’t sensitive enough to detect these enormous placebo effects, and the difference is simply that the placebo-focused trials do a better job of evoking placebo effects. I think this is putting a positive spin on the fact that these placebo-focused trials exploit response bias. Worse, researchers motivated to find a large placebo effect are more likely to engage in questionable research practices in order to produce such an effect. Both factors probably play a role. Preregistered trials seem to have a particularly difficult time establishing a large placebo effect. Roelofs et al. (2000), mentioned in the above quotation, not only specify the randomization procedure in detail, but many other details not present in the other studies, down to the method of data storage and the position subjects were to sit in. These especially careful researchers produced a nonsignificant placebo effect of only half a point on a 100-point scale. As in the rat studies examined below, indications of high research quality seem to correlate with not finding a placebo effect, but high-quality research is so rare that it is difficult to evaluate formally.

Since studies specifically seeking to find a placebo effect were the most fruitful for placebo advocates, the analyses continued. Forsberg et al. (2017), in The Placebo Analgesic Effect in Healthy Individuals and Patients: A Meta-Analysis, in the Journal of Psychosomatic Medicine, find an absurdly large placebo effect on pain, even larger than those of Vase et al. (2002):

The average effect size was 1.24 for healthy individuals and 1.49 for patients. In the studies with patients, the average effect sizes of placebo treatment were 1.73 for experimentally induced pain and 1.05 for clinical pain.

To put these effect sizes into context, Watso et al. (2022) found an effect size of .84 for 5 mg of morphine on experimentally induced pain using a cold pressor test, but no effect of placebo. The same laboratory found an effect size of d = 1.48 for a 75 μg dose of fentanyl on the same pain protocol, while again finding no effect of placebo on either objective or subjective measures. (Again, researchers who do not have their thumbs on the scales to find a placebo effect do not find one.) Placebos in placebo studies are not only more powerful than placebos in controlled trials: they are more powerful than fentanyl. Rather than laughing at these absurd results supporting a powerful placebo effect, many otherwise skeptical people take them seriously, just as they did with behavioral priming studies.

Despite being published in 2017, there is no mention of study quality or quality analysis in Forsberg et al.’s meta-analysis, and concerns of publication bias are minimized. Rather than providing evidence of the placebo effect, I think that the difference between effects found when researchers are and aren’t motivated to find an effect illustrates the power of questionable research practices – and the power of research designs that exploit demand characteristics. We will address the difference in effect sizes between garbage-in-garbage-out meta-analyses and large-scale, preregistered trials in a later section.

On publication bias, Forsberg et al. say:

Visual inspection of the funnel plot for the overall effect revealed some asymmetry, indicating a publication bias with too many small sample studies with large effect sizes and too few small sample studies with small effect sizes. Even if there may be some publication bias, the value of the file drawer statistics indicated that at least 7927 unpublished studies with no effect of placebo treatment would be needed to reduce the placebo analgesic effect to a nonsignificant level. This is considerably higher than the suggested limit (5 K + 10 = 350; 33), and thus, it is unlikely that such a large number of unpublished studies with zero findings should exist.

This type of analysis has always bothered me, as it seems to assume that studies are always produced honestly and that the only issue that can happen is that a study is not published. In reality, it seems that a very common problem is massaging studies until they produce a significant result, which is different from honest non-publication. It seems to me that the numbers would look very different if we acknowledged that many of the studies with large, significant results would have been null if, for example, the analysis had been preregistered and followed. However, when studies are designed to exploit response bias, they may in many cases not even need to mess with their data to produce a result.

One study Forsberg et al. mention is Charron et al. (2006), which studied 16 subjects with low back pain and reported enormous effects of deceptive placebo on self-reported pain (over 20 points in one group of 8 patients). Interestingly, they only got a big effect for low back pain, compared to basically no placebo effect for a cold pressor test (putting a hand in a bucket of cold water). This is in contrast with the findings of Forsberg et al., who found a much larger effect for experimental pain than for clinical pain like chronic low back pain. Even Charron et al.’s sample size of 16 should have been enough to detect an effect as large as 1.73, but methodologies vary and the noise mine is very noisy.

After explaining that the enormous placebo effect dwarfs the effect of accepted real treatment methods, as I have noted above regarding morphine and fentanyl, Forsberg et al. caution:

This does not mean that placebo treatment should be the treatment of choice over other evidence-based pain treatments. Most of the included studies in the present meta-analysis were performed in a laboratory, investigating short-term effects of placebo analgesia. Moreover, whereas studies on treatment for pain commonly use a double-blind design, several of the studies on placebo analgesia used a single-blind design, which most likely increased the effect
of the placebo treatment.

I personally think prescribing placebos to chronic pain sufferers would be cruel, and the fact that people take this idea seriously shows the damage that scientific fuckery can do in the real world, even if it seems harmless (compared to all the fraudulent Alzheimer’s research, for example). Placebo effects in studies like these are large because the research is sketchy, not because placebos are effective.

Forsberg et al. add: “To our knowledge, there is only one study of placebo treatment of long duration outside of the laboratory. This study showed a large analgesic effect across 50 days of placebo treatment in one patient.” (Citation omitted, emphasis mine.) This study was Kupers et a. 2007, Naloxone-insensitive Epidural Placebo Analgesia in a Chronic Pain Patient, which interestingly found no effect of an opioid antagonist on the patient’s placebo response, an issue discussed in a later section.

Subjects involved in an experiment are in a special context very different from everyday life. They are under the impression that the research is important, and wish to play the role of a good experimental subject. “Demand characteristics” reflect a vast repertoire of abilities subjects bring to the experiment. Subjects improvise along with whatever the researchers are up to. Researchers actively trying to find a placebo effect present subjects with a different kind of game to play, compared to researchers testing a treatment. Even without manipulating data, it is possible to produce a spurious “placebo effect” simply because subjects are willing to play along, interpreting cues from researchers and context to figure out the right way to play. Some methodologies exploit this more than others, but by definition, no study comparing placebo to no treatment is blinded.

When talking about scientific-sounding ideas like effect sizes, it is easy to forget what’s really going on underneath the numbers. Subjects may be prompted to report on a survey that they feel better, but to interpret this as subjects actually feeling better is often a mistake. The picture that emerges is that a placebo pill has almost no effect when administered by researchers who do not care about the placebo effect, but the exact same pill has an enormous effect that dwarfs the effect of all existing treatments when administered by a researcher who really wants the placebo effect to be real. The most parsimonious explanation is that it is the research practices, rather than the placebo, that creates the large effect sizes.

The Open-Label Placebo

Placebos have traditionally been administered deceptively, and the deception was thought to be inherent to the efficacy of placebo. If you weren’t told that the pill you were taking was a powerful painkiller, why would you report feeling (slightly) less pain on a survey? But perhaps you have heard that placebos “work” even when you know it’s a placebo, and even if you don’t believe in the placebo effect. This claim comes to us from “open-label” placebo research.

In open-label placebo designs, subjects are told that they are receiving a placebo, and usually given some kind of rationale for why the inert substance should have some effect (contrary claims some researchers have speculated may induce “cognitive dissonance”). In my model in which placebo effects are driven by politeness and roleplaying on the part of subjects, this rationale should be enough to induce them to change their responses on self-report measures, but as with regular placebos, should have no effect on objective measures. This seems to be the case, and suggests that exploiting demand characteristics plays a larger part in apparent open-label placebo effects than other forms of fuckery.

In the meta-analysis of open-label placebo effects of Spille et al. (2023), just as in the Hróbjartsson and Gøtzsche analyses of deceptive placebo, there were significant effects on self-reported instruments, but not for objective outcomes, such as in Mathur et al. (2018), finding no effect of placebo on wound healing (it was almost significant in the wrong direction). Although “the overall quality of the evidence was rated low to very low,” and in my opinion likely exaggerates the efficacy of open-label placebo on even self-report measures (as I will explain), this is exactly what would be expected if the placebo effect were a function of response bias. By definition, an open-label placebo study cannot be blinded. By the nature of the methodology, roleplaying as if the placebo works is invited, not discouraged.

Some of the pain outcomes classified by Spille et al. as “objective” (none of which found any significant effect of open-label placebo) were still under the voluntary control of subjects, such as heat pain thresholds. While none of the “objective” pain measure results here were significant, including the overall meta-analytic effect, many other authors do report a placebo effect on “objectively” measured pain outcomes, as discussed in an earlier section. It seems plausible to me that these can be products of roleplaying, such that I would not conclude that these results are necessarily fraudulent. (Many do, however, seem too large to be real.) An outcome like wound healing, on the other hand, cannot be the product of roleplaying, as long as a blinded observer is rating the wound. The same goes for pregnancy, discussed above. That is why placebos don’t affect these types of outcomes.

Spille et al. give a meta-analytic effect (standard mean difference) of 0.43 (95% CI = 0.28, 0.58) in their sample of non-clinical studies (“20 studies comprising 1201 participants were included, of which 17 studies were eligible for meta-analysis”). Interestingly, in a smaller meta-analysis of clinical samples (“We included k = 11 studies (N = 654 participants) into the meta-analysis”), the same lab (Wernsdorff et al. 2021) reports an enormous meta-analytic effect size of .72 (95% Cl 0.39–1.05). All of Wersndorff et al.’s included studies were self-report measures except one, which found no effect. They note that excluding four studies with high risk of bias, their effect size went down to a still-incredibly-large .49 – though not as large as the hilarious effect sizes reported in Forsberg et al. (2017) for deceptive placebos on pain.

To get an idea of the studies under analysis, let’s look at some of the studies with the largest effect sizes in the Spille et al. (2023) paper. The largest effect size in the self-report category is Guevarra et al. 2020, Placebos without deception reduce self-report and neural measures of emotional distress, a standard mean difference of a whopping .99 between open-label placebo and no treatment on self-reported emotional distress while viewing scary pictures (over twice the effect reported by Schienle et al. (2023), discussed in the next section, in a preregistered trial). Guevarra et al. is also the only study to produce a statistically significant “objective” effect (SMD=.38), and the only study the authors describe as “high risk of bias”(reference 18 in Spille et al. 2023). Guevarra et al.’s “objective” measure, the “neural” measure, is an EEG measure called the sustained late positive potential. I will discuss why this is not actually an objective measure in the later section on brain imaging.

The second-largest effect size in the self-report category was El Brihi et al. 2019, with a SMD of .74. They gave healthy subjects a package of pills with this excellent art and had them take either one or four tablets per day:

Their subjects gamely reported feeling much better on various self-report wellbeing measures, although there was no dose-response relationship for the placebo. Bräscher et al. (2022) attempted to replicate this result (Open-Label Placebo Effects on Psychological and Physical Well-Being: A Conceptual Replication Study), but the replication attempt failed. Unfortunately, their study was reported too late to be included in the Spille et al. meta-analysis. They also used cool art:

Interestingly, one of the polite reasons they give for possibly not replicating the results of El Brihi et al. is that the name “Pharmacebo” reminded participants that they were taking a placebo, although the El Brihi et al. name is similarly suggestive (“Placibax”) and has the word “Placebo” printed on the packaging. I think the most likely reason for the failed replication is that the Bräscher et al. study was of higher quality (for example, assessing symptoms daily instead of once after five days), and the true effect (if we can even speak of a “true effect” made mostly of response bias) is much closer to zero than to .74.

The third-highest effect was Mundt et al. (2017) for lab-induced thermal pain, with a standard mean difference of .69 versus no treatment – but what they found was not decreased pain with placebo, but increased pain in the control group at repeated baseline. Here is their figure:

This seems sketchy to me, even ignoring that p = .045, because usually a placebo effect refers to a decrease in pain. But perhaps sensitization is the expected course for this type of trial, and placebos (open-label or deceptive) prevent sensitization? This seems not to be the case from what I can gather. For example, a study they reference, Chung et al. (2007), using the same methodology and rationale, did not find any such sensitization effect:

Mundt et al. 2017:

A Medoc Thermal Sensory Analyzer (TSA-2001, Ramat Yishai, Israel) was used to deliver all thermal stimuli. Thermal stimuli of 3-s duration were delivered to the ventral forearm via a contact thermode. Temperatures ranged from 43 to 51 °C. Temperature levels were computer controlled by a contractor-contained thermistor with a preset baseline of 32 °C. Stimulation sites were alternated such that no site was stimulated within a 3-min interval to preclude sensitization effects.

Chung et al. 2007:

Medoc Thermal Sensory Analyzer

All thermal stimuli were delivered using a computer-controlled Medoc Thermal Sensory Analyzer (TSA-2001, Ramat Yishai, Israel), which is a peltier-element-based stimulator. The stimuli were a range of temperatures from an adapting temperature of 33°C up to 51°C. Stimuli were applied in a counterbalanced order to the forearm by a contact thermode and were 3 seconds in duration. Multiples sites located on the forearms of both arms were employed. Stimuli presentations were timed such that no site was stimulated with less than a 3-minute interval to avoid sensitization of the site.

Another study, Hollins et al. 2011, using the same skin sites over and over finds the opposite of sensitization (habituation) early on, with later sensitization never going back up to the original level. That is, the rationale of not reusing the same site to preclude sensitization seems backwards:

In a similar vein, yet another study (Jepma et al. 2014) only found sensitization when the sites were switched, again the opposite of the stated rationale. Mundt et al. (2017) looks like noise mining to me, rather than a well-demonstrated placebo effect.

Again, I do not expect the placebo effect (which is to say, the politeness or roleplaying effect) to be zero. It might even be large. Open-label placebo studies seem to attract enthusiastic subjects. Wersndorff et al. say, “Because of the novelty of this kind of treatment, patients seemed to enjoy the treatment and described it as ‘crazy’ according to the intake and exit interviews.” They also say, “The risk for the so-called ‘time lag bias’ is also comparatively high, due to the early state of research in this field. This bias indicated that trials with negative results are published with some delay.” I think that is probably optimistic, given the enormous effects reported in Forsberg et al. (2017), since deceptive placebo research is a more “mature” field. While “time lag bias” was prescient with regard to Bräscher et al. (2022) and Schienle et al. (2023), Benedetti et al. (2023) (discussed in a later section) produced an enormous open-label placebo effect of four points on a ten-point scale! In my opinion, the most likely reason for large open-label placebo effects, as with regular placebo effects, is study sketchiness. Even in the absence of fuckery, when your entire research paradigm exists to exploit demand characteristics, you will indeed produce demand characteristics.

The Placebo and “The Brain”

Brain imaging studies have redressed earlier criticism that placebo effects might merely reflect a response bias.

Elsenbruch and Enck (2015), Placebo effects and their determinants
in gastrointestinal disorders

In the Spille et al. meta-analysis, the only study to produce a significant “objective” effect is Guevarra et al. (2020). These authors find a significant difference between open-label placebo and no treatment on an EEG measure, the late positive potential (specifically the sustained late positive potential measured at between one and six seconds from the stimulus), when subjects view scary pictures. This is their money shot:

They say, importantly, “These results show that non-deceptive placebo effects are not merely a product of response bias.”

The problem here, I think, is that their “objective” measure is in fact entirely possible to produce through response bias. Consider a survey that a subject has filled out. Is this an objective measure? It’s written on paper (or typed into a computer), and can be viewed by a neutral observer. But we wouldn’t call it an objective measure, because it is produced through the voluntary behavior of the subject.

Many authors find that the sustained late positive potential can be influenced intentionally and voluntarily on the part of the subject. Moser et al. (2014) find that, when subjects are looking at scary pictures, the practice of voluntary “cognitive reappraisal” (“participant should imagine that the pictured scene improved and to think of the image in a more positive light so as to decrease the intensity of their negative emotions”) can significantly affect the late positive potential in the exact same way:

Wang et al. (2024) make a similar finding (Watch versus Reappraisal conditions):

Studies find similar magnitudes of difference between late positive potential responses for voluntary “cognitive reappraisal” as Guevarra et al. found for open-label placebos.

On the self-report side, “cognitive reappraisal” might be regarded as a kind of maximal control for response bias, since researchers essentially ask subjects to decrease their ratings voluntarily. In a preregistered trial, Schienle et al. (2023) gave their “cognitive reappraisal” subjects an instruction to voluntarily change how they reacted to scary pictures: “participants were instructed to apply the strategy of cognitive reappraisal in the picture viewing task by imagining that the shown situations and objects are not real, but created by a special effects artist for a Halloween movie.” The authors of course found “regions of interest” in fMRI data, both overlapping with open-label placebo subjects and distinct. But more interesting was the effect on subjective ratings of disgust for the pictures. Compared to the disgust ratings of the “passive viewing” group, the open-label placebo group showed an effect of d = 0.39 (much smaller than Guevarra et al.’s finding of .99), whereas the cognitive reappraisal group showed a massive d = 1.02 reduction in disgust ratings. While the open-label placebo design is subtly suggesting that participants portray themselves as less disgusted (or perhaps less pained, or depressed, or allergic), directly asking people to portray themselves as less disgusted seems to be much more effective in changing survey responses.

If brain imagine measures are generally considered “objective” and not under voluntary control, this seems to imply that people do not have voluntary control over their own brains. I am not sure what metaphysical model this claim is working with. But some brain responses may be under more voluntary control than others, as demonstrated with the late positive potential. This may also be true of fMRI results.

In a large meta-analysis (n=603 participants) of individual participant fMRI data, Zunhammer et al. (2021) find a small (“g < 0.2″) but statistically significant difference in “pain-related brain activity, as compared to the matched control conditions” when combining patient data from both conditioning and “suggestion” (deceptive placebo) studies. (Since they found no difference between conditioning paradigms and normal placebo designs, I won’t go into the distinction here, but will present it in the section on animal models.) The placebo effect in “the brain,” as revealed by fMRI, seems to be scattered around an assortment of different brain regions, with one notable exception.

In an earlier paper analyzing the same 603 participants, the same lab found a small (g = −0.08 [95% CI, −0.15 to −0.01]) but (barely) statistically significant effect of placebo on the “Neurological Pain Signature,” a set of brain areas allegedly activated by pain (Zunhammer et al. 2018). In a “conservative” analysis removing studies rated at high risk of bias, the magnitude of the effect was even smaller, and the 95% CI included zero (g = −0.07, 95% CI, −0.15 to 0.00). Even in the most carefully selected subgroup of “placebo responders” (“which included only participants showing a behavioral placebo response greater than the study median and excluded potentially ineffective placebo treatments and outliers”) that they manage to coax a Bayesian-significant result out of, they note that “effects of placebo on the NPS were only 4% to 14% as large as the overall NPS response to painful stimulation.”

In the 2021 meta-analysis, referring to the study just described (Zunhammer et al. 2018), they say “This previous study revealed that behavioral placebo analgesia was associated with significant but small effects in the NPS, pointing to the relevance of other brain areas and networks.” However, in Botvinik-Nezer et al. (2024), Placebo treatment affects brain systems related to affective and cognitive processes, but not nociceptive pain, a study from still the same lab with almost as many (n=392) participants and a “pre-registered analysis,” no placebo effect on the NPS could be detected. They report that “placebo did not decrease pain-related fMRI activity in brain measures linked to nociceptive pain, including the Neurologic Pain Signature (NPS) and spinothalamic pathway regions, with strong support for null effects in Bayes Factor analyses.”

So whatever is going on with placebo effects on the brain, the one set of areas it doesn’t seem to involve is the set of areas associated with pain perception. The authors conclude, “Our results indicate that cognitive and affective processes primarily drive placebo analgesia.” I am just an ignorant banana, but to me the small effect of placebo on fMRI data almost everywhere but the pain-activated areas suggests that, similar to the late positive potential in EEG, it could be a matter of measuring voluntary effects, or “cognitive and affective processes.”

Interestingly, Spille et al. (2023) seem surprised that they do not get a significant result for “objective” outcomes from open-label placebo. They say, in their discussion section:

Regarding the differences between self-reported and objective outcomes, our finding of a null effect for objective outcomes raises the question of whether OLPs and deceptive placebos have the same pattern of effect, as changes in objective outcomes have been repeatedly demonstrated in studies using deceptive placebos. One might therefore hypothesize that OLPs, unlike deceptive placebos, do not entail biological changes. However, Kaptchuk & Miller emphasize that also deceptive placebos primarily affect self-reported and self-appraised symptoms. Further studies comparing the effects of OLPs with deceptive placebos on objective outcomes are needed to clarify this issue.

The claim “changes in objective outcomes have been repeatedly demonstrated in studies using deceptive placebos” might seem surprising after reviewing the clinical trial data from the Hróbjartsson and Gøtzsche meta-analyses (even Forsberg et al. do not claim to find a placebo effect on objective outcomes). But Spille et al. cite for the claim a paper by Fabrizio Benedetti, a great placebo believer and advocate for placebo research and a figure whose research we will learn more about in the opioid antagonist section. The paper, published in 2012 in the Journal of Acupuncture and Meridian Studies, is called Placebo-Induced Improvements: How Therapeutic Rituals Affect the Patient’s Brain. I am surprised by this choice, not because I don’t trust the Journal of Acupuncture and Meridian Studies, but because I would have imagined a series of meta-analyses would provide more certainty of evidence. Benedetti (2012) makes many interesting claims, and the word “objective” does not occur in the document, so it is up to interpretation. His laboratory sometimes finds an effect of placebo on “objective” measures of pain, discussed above, such as time that ischemic arm pain is tolerated. As I have explained, these results may also be a product of roleplaying and politeness, although the results are often so very large as to seem suspicious, and other laboratories seem to have difficulty replicating these effects. The other results he presents that might be interpreted as “objective” outcomes are concerned with measures of chemicals like endogenous opioids and dopamine (as well as a reference to an fMRI study of placebo acupuncture). This is a major focal point of placebo belief, particularly the claim that we “know” the placebo effect is an “objective” phenomenon because it involves the release of e.g. endogenous opioids, and we can abolish the placebo effect with a hidden injection of e.g. opioid antagonists. I will review the evidence for these claims in the next section.

Opioid Antagonists (And Friends)

One of the most understandable bases for the placebo belief is the idea that the placebo effect is objective and measurable, and not response bias or woo, because it is based on endogenous opioids (or similar chemicals, perhaps dopamine), and we can experimentally extinguish the placebo effect by giving a hidden injection of an opioid antagonist like naloxone. This cannot be response bias, because subjects are not aware of whether they are receiving naloxone or saline, and their subsequent pain ratings differentiate the two. This offers the promise of a truly blinded demonstration of a placebo effect.

This idea stems from the work of two important figures: Jon Levine, the pioneering neuroscientist who along with coauthors Newton Gordon and Howard Fields invented the idea, and their former student, Fabrizio Benedetti, referenced in the last section. Almost all of the research confirming the effect comes from these two sources, and other laboratories have found it difficult to reproduce these results. The majority of studies included in a twenty-year-old meta-analysis of the alleged effect, Endogenous opiates and the placebo effect: A meta-analytic review, by Sauro and Greenberg (2005), are from the laboratories of Levine or Benedetti.

The Sauro and Greenberg (2005) meta-analysis is old enough that it doesn’t include an analysis of publication bias or statements like “Study quality ranged from abysmal to hilarious,” but it doesn’t seem to have been updated in the age of open science.

The first thing that stands out from the meta-analysis is the absolutely massive estimated meta-analytic effect of placebo, d+=0.89 (95% CI 0.74 –1.04). This is even larger than the effects reported in the open-label placebo meta-analyses of various outcomes, which in turn were so much larger than the estimate of .28 from Hróbjartsson and Gøtzsche on self-reported pain (though again still not as large as the estimates in Forsberg et al., 2017, with which it has many overlapping studies). These researchers have access to very powerful placebos indeed! Some subgroup effects are even larger, with the largest being d+= 1.23 for the placebo effect on tourniquet-induced ischemic pain. The meta-analytic effect of naloxone in reducing the placebo effect is almost as massive, at d+=0.55 (95% CI 0.39–0.72). Note that naloxone no longer “abolishes” or “extinguishes” the effect, but merely reduces it somewhat in this analysis. The largest subgroup effect is an incredible d+= 1.37 for the reduction of placebo effect by naloxone in capsaicin-induced experimental pain (based on two studies, both reported in Benedetti et al., 1999).

Of the twelve papers included in the meta-analysis, seven have Levine or Benedetti as an author, and all seven of those produce positive results supporting all hypotheses. This is a bit surprising, since quick use of a couple of internet calculators suggests that to detect an effect size of .55 at 80% power would require a sample size of at least 84 subjects with 42 in each arm, assuming a one-tailed hypothesis. However, as detailed below, modern studies that perform a power analysis only find the need for the same old 13 or 14 subjects per arm, I think because they assume a much larger effect size, perhaps an effect size even larger than the placebo effect itself, so what do I know. Almost all of these studies have 11 to 17 subjects per arm, and the largest, Benedetti et al. (1999) with 24 to 29 subjects per relevant arm, gets a significant result in every comparison. (See appendix for group counts in all studies, and for a note on the alleged significance of one result.)

Benedetti and Levine get a positive result every time using an eclectic variety of methodologies and measures and surprisingly small sample sizes. On the other hand, of the five studies not conducted by Levine or Benedetti, only one gets a positive result, comparing a group of 16 to a group of 14. Some laboratories have all the luck!

I have briefly summarized every study from the Sauro and Greenberg meta-analysis with basic study characteristics and findings, but since my summary amounts to over 4000 words, I have turned it into an appendix for the curious.

As far as I can tell, there has never been a preregistered or multi-center or Manylabs-style replication attempt for the effect claimed in Sauro and Greenberg (2005). There is a preregistered study from this year, Dopamine has no direct causal role in the formation of treatment expectations and placebo analgesia in humans (Kunkel et al. 2024), which even included a power analysis indicating that 165 subjects would be needed (55 per group), and managed to get 168. The authors, as stated in the title, find no evidence for either dopamine agonists or antagonists in the formation of the placebo response in a conditioning paradigm. But I can’t find anything like this for opioid antagonists.

In the modern era, a recent study (Pontén et al. 2020) that looked at pain ratings and fMRI data in a conditioning paradigm failed to find any effect of naloxone on open-label placebo analgesia (for painful pressure applied to the thumbnail) in either measure reported. They performed a power analysis that justified a small sample size:

An a priori power analysis was performed to determine the sample size required to detect a pain-cue effect (n = 13) based on a previous data set with similar design. Calculations were performed in G*Power (3.1) based on differences in pain ratings (0–100) between two pain cues (dependent means) M = 17, SD of difference = 13, alpha = .05, power (1 − β) =.99, two-tailed.

Benedetti et al. (2023), however, as always find an enormous positive result (in “placebo responders”) in reversing the placebo effect of open-label placebos in ischemic arm pain. (The placebo effect that they produce is a whiplash-inducing d = 7.29 in “placebo responders,” off the charts even in the world of Forsberg et al. (2017).) They also perform a power analysis justifying an almost identical sample size:

An a priori analysis of power and sample size was performed regarding the expected difference between saline (group 6) and naloxone (group 7). A sample size of 13 was calculated by setting the desired power at 0.8, P at 0.05, the expected difference between naloxone and saline at 1 or 2, and the expected variability at 1 or 2 SD. Therefore, we decided to test a sample of 14 subjects for group 6 and 14 subjects for group 7.

What could explain the differing results? The pain stimulus was different between the two studies, so perhaps only the placebo effect in ischemic arm pain is affected by naloxone. Pontén et al. did not select their subjects on the basis of being in the dubious group of “placebo responders” (although they did demonstrate a conditioned placebo effect), while Benedetti et al. did. Pontén et al. recruited subjects based on an “advertisement,” whereas Benedetti et al. recruited students and employees at the university and laboratory where the study was conducted. Pontén et al. used a conditioning paradigm and Benedetti et all. used a suggestion paradigm, but Amanzio and Benedetti (1999, discussed in appendix) previously claimed to demonstrate the effect with a conditioning paradigm with similar group sizes.

It’s interesting to look at the Benedetti lab results for deceptive placebo back in 1996:

The effects of naloxone vs. saline amount to less than two points on a ten-point scale and occur mostly 20 to 25 minutes after injection of naloxone in the 1996 deceptive placebo study. On the other hand, the open-label naloxone results from 2023 are larger and occur much earlier. The “placebo effect” of deceptive vs. open-label placebo appears almost identical in magnitude (absolutely massive), but occurs earlier in the 2023 study. Naloxone causes pain ratings to return to baseline much earlier, too, and pain goes up beyond baseline, even though in 2023 the hand squeeze device required less pressure to close (5 kg vs. 6.5 kg) and they used a lower cuff pressure to induce ischemia (200 vs. 250 mm Hg):

It is surprising that naloxone would be so much faster and more effective at reducing the effect of an open-label placebo compared to a deceptive placebo. (In both cases, the relevant groups were constructed of “placebo responders” and the experiment was started when a pain rating of 7 was reached.)

Benedetti et al. (2023), as in all their studies, find no effect of naloxone on pain in the absence of a placebo effect to “abolish.” A recent meta-analysis (Jensen 2021) found a small effect (g= .23) of naloxone on pain in general, but the author notes that “there was considerable heterogeneity present,” and “due to reporting bias in the literature, the size of this effect may be overstated.” It seems odd that if the endogenous opioid system is involved in pain perception, that opioid blockade should have such a small and unreliable effect with regard to various kinds of pain and pain modulation, yet somehow specifically act to reduce the “placebo effect.” This will be expanded on later in this section.

Just recently, Dean et al. (2024) published a preregistered study on the effects of naloxone on placebo analgesia (sham mindfulness) in men and women. With a similar sample size (15 per group), they found an effect in men but not in women, contrary to both Pontén et al. 2020 (whose subjects were all men but did not get a result) and Benedetti et al. (2023) (who found no difference between men and women). In the female subjects in Dean et al. (2024), naloxone actually increased their placebo analgesia, though not significantly. They found a much higher placebo effect in men than in women, contrary to the common finding in both randomized controlled trials and placebo experiments that there is no difference (e.g. Enck and Klosterhalfen, 2019). The hypothesis that researchers are still mining noise cannot be rejected.

I turn now to a 2015 systematic review, examining more studies on the effect of naloxone on pain modulation in various contexts, and visit some contemporary research in this area, to investigate the foundations of this alleged effect.

The review is Endogenous Opioid Antagonism in Physiological Experimental Pain Models: A Systematic Review, by Werner et al. (2015). These authors are not simply focused on the placebo effect, but on the effects of naloxone-type drugs on various types of experimental pain and pain reduction (or increase). They lead their conclusion with a quotation from a 1978 study by Grevert:

The consistent failure to find an effect of naloxone on experimental pain in humans suggests that endorphin release did not occur during these procedures

Werner et al. conclude:

This systematic review on endogenous opioid antagonism in physiological experimental pain models concludes that naloxone appears to have a demonstrable and relatively reliable effect in stress-induced analgesia (in all 7 studies) and repetitive transcranial magnetic stimulation (in all 3 studies). In all other pain models, both naloxone and naltrexone demonstrate a variable and unreliable effect.

Not a very reassuring conclusion! This review looks at studies that attempt to manipulate pain in various ways, either inhibitory (decreasing pain, as with placebo) or sensitizing (increasing pain, as with nocebo), and measure whether naloxone has any effect on the pain ratings. For example, in the seven “stress-induced analgesia” studies mentioned positively in the conclusion (all published between 1980 and 1986), subjects are subjected to pain after being stressed out by being made to do math problems or a public speaking task. After this treatment, but not after control non-treatment, they report lower levels of pain – stress reduces self-reported pain! Opioid antagonists, we are told, reduce this pain relief. Similarly, a few authors claim they can reduce pain with transcranial magnetic stimulation, and that this pain reduction is reversible by opioid antagonists. One study looked at the effects of distance running on laboratory-induced pain, and found that naloxone reversed it for ischemic (tourniquet) pain, but not for heat pain. This is emblematic of the entire enterprise: a mishmash of no result, some result, a result in the opposite direction, etc.

Other studies examine whether naloxone-like drugs affect pain ratings, thresholds, and sensitivity in general, again with widely varying results. One body of research focuses on the inhibition of pain with more pain (of a different type or in a different area, such as having one foot in a bucket of freezing cold water while experiencing laboratory-induced heat pain on your arm). This is called “conditioned pain modulation” (CPM). Does naloxone reduce this modulation? One study (King et al. 2013) produces a summary of studies that perfectly encapsulates the research paradigm, in both sample size and variability of results:

In this paradigm and others, the studies are all over the place, revealing no consistent effect of opioid antagonists on pain or pain modulation (except where exogenous opioids are concerned, which they do seem to effectively and reliably reverse, just as they seem to cure people who are overdosing). What about the two paradigms in which naloxone seems to have a “relatively reliable” effect, stress-induced analgesia and transcranial magnetic stimulation?

I mentioned above that all of the stress-induced analgesia studies that show a reverse of analgesia with naloxone are from the early-to-mid 1980s. Scientific practices have changed somewhat since then. What is up with current research?

A study from 2022, al’Absi et al. (2021), got a small effect at p = .04, the Cursed Value, for reduction of stress-induced analgesia on a cold pain test (subjects had to hold their hands in a bucket of freezing ice water slurry as long as they could stand). But they found no effect in a heat pain paradigm (“thermal stimulation device”), and Bruehl et al. (2022), same lab, replicated the null result for the heat pain paradigm. Apparently the involvement of endogenous opioids in stress-induced analgesia is not so “relatively reliable.”

As for the three transcranial magnetic stimulation studies, it is possible that TMS is somehow the only method of reducing pain that is reliably blocked by naloxone, but I think that is unlikely as TMS is one of the fakest areas of research outside of psi and telekinesis.

Until there is a large, pre-registered multi-center replication attempt, preferably not led by researchers with high allegiance to the effect who produce positive results over and over with tiny sample sizes, it seems that the evidentiary basis for a placebo effect modulated by endogenous opioids is not very solid.

Unless?

I have implied that I think something sketchy is up with the research on the effects of opioid antagonists on placebo analgesia, given the inconsistency of findings. However, I think there are two possible explanations that would allow the studies finding an effect to be perfectly honest, although still meaningless.

The first is that the administration of opioid antagonists (vs. placebo) may not be blind. That is, subjects may be able to detect that they have received an opioid antagonist, because opioid antagonists are unpleasant. For example, Wardle et al. (2016) found that their subjects were able to feel the effects of both a 25 mg and a 50 mg dose of naltrexone, differentiating it from placebo on all three measured indicators, “feel drug,” “dislike drug,” and “fatigue.” Naltrexone also caused nausea, with “near 0%” of subjects in the placebo condition reporting nausea, while 24% and 35% reported nausea in the 25 mg and 50 mg naltrexone conditions, respectively. This is in contrast with the usual method of confirming blind, which is to assess whether subjects guess which condition they are in, and which is usually found not to differ from chance (e.g. Inagaki et al. 2016). This may simply be a crappy method of assessing blind, as subjects may feel subtly worse but not attribute it to the drug, which presumably few subjects have ever taken before.

This was also the finding of Schull et al. (1981) for naloxone, who found that naloxone increased both the intensity and unpleasantness of the kind of ischemic arm pain that Benedetti et al. usually employ. Subjects given naloxone rated their pain as more intense and tolerated it for less time, and also rated their mood worse. The difference was large, similar in magnitude to the placebo effect reductions that Benedetti et al. usually find.

The second is a weirder possibility, which as far as I can tell has not been investigated. What if the effect of opioid antagonists is to make subjects less interested at playing along with researchers in general? For example, Rütgen et al. (2015) got an effect for reducing placebo analgesia with naltrexone, but got an even larger result on the effect of naltrexone on subjects’ imagined pain ratings for a confederate:

Peciña et al. (2021) found that naltrexone somewhat reduced the reported placebo effects of a placebo said to be a fast-acting antidepressant. Chelnokova et al. (2014) found that naltrexone reduced male subjects’ button pressing to keep an attractive face on the screen. These results are consistent with opioid antagonists generally reducing playing along with researchers (although they are also consistent with opioid antagonists making subjects feel worse in general). While I still suspect that all these results are just noise, it would be worth differentiating these possibilities. What if there was a drug that reduced response bias?

Mind Cure for Mice? Animal Models of Placebo Analgesia

Since animals, unlike bananas, can’t use language in the way humans do, they can’t be treated with placebo by suggestion (for instance, by telling them that an injection is fentanyl when it is actually saline, or giving them a rationale for an open-label placebo). Many of them can, however, learn, so animal models of placebo analgesia rely on conditioning paradigms.

In a typical placebo study, a subject might be given a pill or cream and told it is a powerful pain reliever, whereas a control subject might be told that it is an inert pill or cream (or no pill or cream at all may be given, as in the control arms, really control hands and feet, in Benedetti et al. 1999). In a conditioning study, subjects are trained and kind of gaslit with a conditioning procedure. For example, let’s imagine the placebo is a green light. The subjects are receiving electric shocks and asked to rate their pain. During the conditioning phase, they are given less intense, less painful electric shocks when the green light is on, and more intense, more painful shocks when the green light is off. They are told that the shocks are objectively of the same intensity. Basically, they are trained that green light means less pain. Afterwards, in the testing phase, they briefly guess wrong – specifically, they report less pain when the green light is on, at least until they realize that it no longer has any meaning. This is certainly deceptive, but I’m not convinced that this type of “conditioning” is what people mean when they say the placebo effect is real. Nonetheless, it is a common paradigm, because it is pretty easy to get subjects to essentially guess wrong, at least for a little while (although many authors fail to replicate this). Animals may be capable of being trained to “guess wrong” too.

One reason it is difficult to measure a placebo effect in animals is that it is difficult enough to measure pain in animals in the first place. Obviously, they can’t rate their pain on a scale of 1 to 100. Wodarski et al. (2016) attempt to bring the field into the age of open science with their Cross-centre replication of suppressed burrowing behaviour as an ethologically relevant pain outcome measure in the rat: a prospective multicentre study. They do not attempt to condition or demonstrate a placebo effect, but simply to replicate a single measure of pain in the rat, which is a reduced degree of burrowing. They did in fact verify that burrowing was suppressed (measured in grams of material displaced) in seven of eleven of the included studies. One interesting problem they identify is that blinding was impossible to maintain, as the substance they used to induce pain was yellowish and viscous compared to saline, such that “allocation concealment could be maintained only in 2 studies.” I applaud their effort, and precisely their honesty and attention to detail highlights how easy it might be to cheat in animal studies (without even going as far as making up data or excluding subjects based on a gut feeling).

Learned analgesic cue association and placebo analgesia in rats: an empirical assessment of an animal model and meta-analysis, a master’s thesis by Swanton (2020), provides a starting point for animal placebo analgesia by helpfully conducting a meta-analysis. Swanton also details the process of conducting three replication attempts of a conditioned placebo effect in rats, offering more detail than is typically seen in scientific papers (e.g. she reports, “Attempts were made to contact the authors [of the replication target study] for further information, but there was no response.”). It is one of the most interesting scientific documents I’ve encountered on this topic, and that is saying a lot. Swanton makes three increasingly valiant attempts (with 15 or 16 rats per comparison group, a large sample by the literature’s standards) to replicate Lee et al. (2014), A new animal model of placebo analgesia: involvement of the dopaminergic system in reward learning. (Fascinatingly, in addition to demonstrating placebo analgesia, Lee et al. claim both the behavioral indicators and laboratory tests of biomarkers were “blocked by a dopamine antagonist but not by an opioid antagonist.” This is in contrast with Kunkel et al. 2024, referenced in the above section, an adequately-powered study that ruled out a role of dopamine in human conditioned placebo responses, and also in contrast with the claims in the Sauro and Greenberg meta-analysis claiming an effect of opioid antagonists.)

Swanton was not able to demonstrate a conditioned placebo response at all, even when going to extremes adding high-visibility visual cues, sound cues, and scent cues to her setup to enhance learning:

After this triple disappointment, Swanton delivers a meta-analysis, noting that there is considerable variability within research paradigms, such that “no two protocols are the same in regards to cue type.” Most studies use a hot plate as an apparatus for pain, modeling the amount of time it takes a rat to withdraw a hind paw as a measure of placebo analgesia (or guessing wrong, in my model), or how much they lick their front paws. There are many other paradigms, as we have already seen with the injected pain-inducing substances, and one team even employed “a model of irritable bowel syndrome that employs an inflated balloon to mimic bowl expansion.” Sometimes rats are conditioned (or not) with morphine, and other times, as here, they are conditioned (or not) with the expectation that the temperature of the hot plate will be lower if certain cues are present.

You might wonder, as I did, why “conditioning” with a drug would result in a placebo effect when an inert substance is substituted, rather than increased pain. (If you drink decaf coffee expecting it to be real coffee, you might feel sleepier, because your body is preparing to deal with a bunch of caffeine that doesn’t come.) Apparently this is a relevant question, and Swanton says that “Early drug conditioning research established that cue-associated morphine resulted in hyperalgesia from drug tolerance, not placebo analgesia,” and “More recent work using almost identical conditioning models have reported opposing results of placebo analgesia.”

21 papers met Swanton’s inclusion criteria for the meta-analysis. Almost all used drugs for conditioning, mostly morphine. Most studies had major quality issues. Almost half of the “main outcomes” of the included studies found no effect (31 out of 65 reported main outcomes). Some studies that did get an effect reported effects as large as a Hedge’s g of 5 (yes, five, not .5). From eyeballing the chart of study quality problems, there was only one study with few quality or blinding issues, Akintola et al. (2019), and it got a null result. Despite these obvious problems with replicability, the overall meta-analytic effect for rodent placebo analgesia was a massive g=0.842, which Swanton drily describes as “consistent with a recent meta analysis in human placebo analgesic effects (Forsberg et al., 2017) that demonstrated a high effect size in people.” There was, however, “extremely significant” heterogeneity, unsurprisingly.

Swanton says:

This is the first time rodent models of placebo analgesia have been meta analysed and is thus the most compelling evidence we have to date that animals are capable of experiencing placebo analgesia.

It’s amusing to me that the “most compelling evidence” comes on the back of multiple apparently sincere and determined but failed replication attempts. This contrast is also seen in other areas of research, as illustrated in Kvarven et al. (2020), Comparing meta-analyses and preregistered multiple-laboratory replication projects. They say:

We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results.

To put it less politely, when a meta-analysis of garbage studies conducted by researchers with their thumbs on the scale is compared to more honest study designs, the more honest designs produce smaller effects, or no effect at all. If these massive effect sizes are real, it is amazing that the researchers with the best study designs should fail to detect them.

The F Word

I want to discuss something that I have alluded to but not addressed as a separate matter: fuckery. There are more kinds of fuckery than can be listed or perhaps even known, engaged in by anyone from a principle investigator to a lowly research assistant, and when fuckery is discovered, it is often impossible to pinpoint who committed it. I am referring to behaviors such as:

• exploiting unblinding by exaggerating measures
• excluding data based on a “gut feeling” or because it wrecks the result
• reporting only results that support the hypothesis
• Wansinking
• continuing to gather data until the result is significant and stopping when it is
• measuring the outcome before the manipulation that’s supposed to cause it happens, but still producing a significant effect
• trying a bunch of different variations of a measure until one works out
• manipulating or faking data

I believe that finding a large placebo effect (or similar mind-cure effect) is a reliable marker of fuckery. Researchers associated with fuckery seem to produce enormous placebo effects, and the same goes for research fields such as marketing. For example, Shiv, Carmon, and, relevantly, Dan Ariely (2005), in Placebo Effects of Marketing Actions: Consumers May Get What They Pay For, produced a large nocebo effect on an objective cognitive outcome (solving word jumble puzzles) from the tiny manipulation of whether the fine print of the study materials said that the energy drink provided to recipients was purchased “at a discount as part of an institutional purchase.” This was published in a marketing journal, the Journal of Marketing Research. Discount energy drinks made the subjects much stupider:

This is in contrast to e.g. a preregistered trial by Kleine-Borgmann et al. (2021) which found no effect of an open-label placebo on test performance, and another preregistered trial by Hartmann et al. (2023) that found no open-label placebo effect on cognitive measures.

Another influential placebo study is Commercial Features of Placebo
and Therapeutic Efficacy, a marketing study somehow published in JAMA, again with Dan Ariely as an author. These authors find extremely large placebo effects of 15-30 points on a 100-point scale, in a painful electric shock experiment that was not approved by the Institutional Review Board prior to execution, resulting in Ariely being suspended from MIT. If you cut corners on IRB paperwork, what other corners are you cutting?

In the closest replication attempt conducted, Tang et al. (2013) found much more plausible placebo effects of no effect, one point, and three points on a 100-point scale at different shock intensities under conditions similar to the above study.

The above Ariely study, by the way, while only a “research letter,” is one of the most influential studies for the claim that expensive placebos work better than cheap placebos. Many are also convinced that properties of a placebo other than price affect their healing value, but the evidence for this is also weak. For example, Meissner et al. (2018), Are Blue Pills Better Than Green? How Treatment Features Modulate Placebo Effects, provide a review of the evidence for this idea. The studies do not evaluate efficacy at all; instead, they simply ask subjects on surveys to associate colors with different possible treatment effects of drugs. Linde et al. (2010) found that needles were more effective placebos than other types, although they note that “Due to the heterogeneity of the trials included and the indirect comparison our results must be interpreted with caution.” Most of the studies that use needles as a placebo are acupuncture studies, an area in which many researchers are especially prone to fuckery.

While not precisely a placebo effect, but still a mind-cure effect, Ronald Grossarth-Maticek and Hans Eysenck published a great deal of research, now mostly retracted because of fraud, claiming that they could reduce cancer incidence in people with “cancer-prone personalities” by providing talk therapy, e.g.:

In the exquisitely-titled What a wonderful world it would be: a reanalysis of some of the work of Grossarth-Maticek, Van der Ploeg (1991) reveals an amusing way in which he caught their fraud: they had provided data years earlier with the identifying details redacted with marker ink, but the ink faded over the years, allowing Van der Ploeg to check the study participants against death records. Gossarth-Maticek, faced with proof of his deception, responded that, among other things, he was simply testing Van der Ploeg’s honesty and wanted to know if he would reveal patient identities when the marker ink faded.

Ellen Langer (in Crum & Langer, 2007, Mind-Set Matters: Exercise and the Placebo Effect) found large placebo effects on weight loss after subjects were merely told that their work (as hotel maids) was good exercise. I have already linked to Gelman and Brown’s (2024) extensive criticism of this study, but this is not Langer’s only instance of egregious fuckery. In Eminent Harvard psychologist, mother of positive psychology, New Age quack? (2014), James Coyne notes that she also advocates mind cures for cancer, much like Grossarth-Maticek and Eysenck. She also published a study (Rodin and Langer, 1977, Long-term effects of a control-relevant intervention with the institutionalized aged.) claiming that giving institutionalized old people a plant to be responsible for and water reduced mortality from 30% in the control group to only 15% in the treatment group. Coyne points out that an erratum published a year after the study essentially retracted the result. Langer is still attempting lucrative mind cures for cancer. When you’re a Harvard psychologist, they let you do it.

In Brooks et al. (2016), Don’t stop believing: Rituals improve performance by decreasing anxiety, now retracted, a paper with disgraced Harvard psychologist Francesca Gino as a co-author, a placebo effect of a placebo “ritual” was found on objective math test performance:

Please count out loud slowly up to 10 from 0, then count back down to 0. You should say each number out loud and write each number on the piece of paper in front of you as you say it. You may use the entire paper. Sprinkle salt on your paper. Crinkle up your paper. Throw your paper in the trash.

For the math study, Study 4, the following explanation is given in the retraction notice:

A reanalysis of the data showed that 11 participants’ datapoints were dropped prior to analysis but their removal was not reported in the paper. The authors report that the decisions to drop data were based on RAs’ written notes. The reanalysis shows that the focal effect becomes non-significant once all participants are included.

To repeat my earlier maxim, if there is a statistically significant effect of placebo on an objective outcome, chances are it is either noise, fraud, questionable research practices, or a mischaracterization of a subjective outcome as objective. Here it was fuckery. This is a rare situation in which the truth was revealed. Unfortunately, for most of these bogus studies, we will never know the truth about how their implausible results were constructed.

Conclusion

Initially, the “placebo effect” appeared powerful because natural improvement (the episodic nature of conditions, regression to mean, or healing over time) was misinterpreted as an effect of suggestion. When placebo arms of controlled trials of treatments were compared to no-treatment arms, it was revealed that the effect of placebo was modest, if it existed at all, and was exclusively a function of subjective outcome measures (response bias or roleplaying). Studies designed specifically to exploit response bias in measuring a “placebo effect” were often able to produce large differences on self-reported outcomes, but never on objective outcomes like wound healing, pregnancy after IVF treatment, or any outcome measured by a laboratory test. Brain imaging studies seem to have confirmed, rather than refuted, the claim that the “placebo effect” is a phenomenon of response bias. Studies finding similar efficacy for “open-label placebos” lend more support to the conclusion that response bias drives placebo effects.

Animal models attempting to demonstrate a placebo effect in animals using a conditioning paradigm suffer from poor replicability, with studies with few quality or blinding issues finding no effect. Studies attempting to prove that the “placebo effect” involves the endogenous opioid system also suffer from poor replicability, with conflicting results from different laboratories. No large multi-center preregistered trial has confirmed the effect, although one such trial produced strong evidence against the involvement of the dopaminergic system. Related research on pain modulation casts doubt on whether the endogenous opioid system is involved with psychological pain modulation at all.

Large placebo and “mind-cure” effects are a common feature of research now known to be fraudulent. The available evidence supports a conclusion that the “placebo effect” is not a real healing effect, but a product of response bias, questionable research practices, and misunderstanding. Placebos only appear to have efficacy to the extent that subjects are encouraged to role-play as if they are effective, and any difference on self-report measures reflects roleplaying, not healing. Inert substances are in fact inert, and are not rendered effective by suggestion. The power of the placebo is to blind subjects and researchers in blinded research designs. The magnitude of “placebo effects” in necessarily-unblinded placebo-focused studies demonstrates the necessity of blinding, not a placebo effect.

Brief review of every study in Sauro and Greenberg (2005)

The first study is Neuropharmacological Dissection of Placebo Analgesia: Expectation-Activated Opioid Systems versus Conditioning-Activated Specific Subsystems (Amanzio and Benedetti, 1999). This, like many of the studies here, is a conditioning study, so it’s worth taking some time to explain what differentiates a conditioning study from a normal placebo study.

In a typical placebo study, a subject might be given a pill or cream and told it is a powerful pain reliever, whereas a control subject might be told that it is an inert pill or cream. In a conditioning study, subjects are trained and kind of gaslit with a conditioning procedure. For example, let’s imagine the placebo is a green light. The subjects are receiving electric shocks and asked to rate their pain. During the conditioning phase, they are given less intense, less painful electric shocks when the green light is on, and more intense, more painful shocks when the green light is off. They are told that the shocks are objectively of the same intensity. Basically, they are trained that green light means less pain. Afterwards, in the testing phase, they briefly guess wrong – specifically, they report less pain when the green light is on, at least until they realize that it no longer has any meaning. This is certainly deceptive, but I’m not convinced that this type of “conditioning” is what people mean when they say the placebo effect is real. Nonetheless, it is a common paradigm, because it is pretty easy to get subjects to essentially guess wrong, at least for a little while.

In Amanzio and Benedetti (1999), as in many of these studies, subjects were conditioned not with a green light, but with literal morphine (or, in a different arm, with ketorolac, a non-opioid NSAID pain reliever). The researchers induced ischemic pain in subjects with a tourniquet and had subjects squeeze and hand exerciser over and over until they could no longer bear it, and subjects given morphine unsurprisingly could bear it for longer than they had when given nothing. The equivalent of the “green light” was the injection, indicating that morphine had been given. The conditioning continued for two days. Then, on the third day, some subjects received an injection of saline, but were told it was morphine, and were able to last somewhat longer than baseline in the ischemic pain test (though not, it seems, as long as with real morphine). The authors claim this worked even when the subjects were told it was an antibiotic, not morphine, though not as much. The next day, there was no injection, and pain tolerance returned to baseline. (That is, subjects guessed correctly again.)

However, if instead of a saline injection said to be morphine, subjects were given a naloxone injection, their morphine conditioning did not work, according to these authors, and pain tolerance again went back to baseline. When conditioned with the non-opioid drug ketorolac, however, the (conditioned) placebo response was not extinguished with naloxone, and subjects could squeeze the hand exerciser as long as they could with a saline injection. Here are the graphs for morphine conditioning – baseline is day 1 (no treatment), days 2 and 3 are morphine conditioning (they squeeze the squeezer for a long time), day 4 is either saline or naloxone described as either morphine (+ expectation) or an antibiotic (without expectation), and day 5 is baseline:

And here are the graphs for ketorolac, with very little apparent effect of naloxone vs. saline:

In some ways, this is a more interesting finding that simply “the placebo effect is about endogenous opioids.” This experiment purports to find two separate kinds of placebo effects, depending on the drug used for conditioning, one destroyed by opioid antagonists, and the other remaining perfectly intact! This does not seem like evidence for a general role of endogenous opioids in “the placebo effect.”

Another finding in this many, many-armed study is that hidden injections of naloxone in the absence of conditioning do not affect pain tolerance. That is, whatever endogenous opioids may be produced in response to pain naturally, naloxone does not seem to affect them – only placebo-generated endogenous opioids?? This finding has been somewhat contested; some authors have even found that opioid antagonists cause “paradoxical” pain relief!

The second study is Amanzio et al. 2001, Response variability to analgesics: a role for non-specific activation of endogenous opioids, also with Benedetti as a co-author. The authors seem to throw the results of the previous study in the garbage, now attempting to show that endogenous opioids are involved in pain relief from ketorolac, too. With a study design that I personally find very tedious, they start with a population of clinical pain patients recovering from thoracic surgery. They found that the dose of drugs needed to achieve a 50% reduction in self-reported pain was much higher when the injection was open (in view of the patient), rather than hidden. As someone who doubts that pain intensity reports are a pure measure of pain experience, and have at least some component of politeness, roleplaying, and judgment of situational appropriateness, it is not very surprising that patients who are aware they have been treated report less pain that patients who are not aware of any treatment. These authors interpret this as part of the placebo effect. Specifically, they say, “the difference between open and hidden [doses to relieve 50% of pain] may be taken as a measure of the placebo effect.”

They also focus on the finding that the dose needed to achieve 50% pain relief was not only larger, but less variable in the hidden injection groups. They then introduce an experimental group of healthy subjects, who are again going to get various injections and wear a tourniquet while squeezing the hand exerciser until they can’t stand it anymore. Here are their results:

They are focusing on the difference between open ketorolac (the highest squeezy test times, at about twenty minutes, a little lower than their previous groups) and the three conditions to the right. Their claim now is that naloxone reduces the “placebo effect” component of the open ketorolac injection. They emphasize the small (but apparently statistically significant, with only 15 subjects per arm) difference between the open ketorolac and the hidden ketorolac, and say that “the effects of a hidden injection of ketorolac and an
open injection of ketorolac plus naloxone were exactly the same.” (Although I must say, visually, open ketorolac plus naloxone looks about in between the open ketorolac and hidden ketorolac.) The addition of naloxone seems to remove the difference between the open and hidden injections in their sample, thereby, in this tedious and roundabout way, demonstrating that it extinguished the placebo effect component of a non-opioid drug.

They also show that their subjects’ time in the tourniquet varied more in the open ketorolac-no naloxone condition than in the other conditions, suggesting that part of the variance is placebo effect, and naloxone removes this:

The third paper is an earlier paper by Benedetti as the sole author, The opposite effects of the opiate antagonist naloxone and the cholecystokinin antagonist proglumide on placebo analgesia (1996). Subjects again have their forearms wrapped in painful tourniquets and squeeze a hand squeezer until they can’t stand it anymore (which, I might add, is a voluntary act), although here subjects are also rating their pain intensity (a more typical self-report measure).

Interestingly, they construct a group of “placebo responders,” a procedure criticized in the section of the main post called Run In, Wash Out. They say: “In order to obtain 60 placebo responders, the remaining 223 subjects received an injection of saline in full view (placebo) when [a pain intensity rating of] 7 was reached. If pain rating was 7 or less at 15 min following the open saline injection, a subject was considered a placebo responder….Most of the subjects (163 out of 223) did not respond to placebo.” There are also a separate 117 subjects assigned to control groups. These subjects are divided between twelve separate experimental conditions, with between 10 and 67 subjects per condition.

In this experiment, among two of many other groups of subjects, an injection of saline (that subjects are told is a potent painkiller) is given at the start of the squeezy test when pain reaches a 7 (which typically takes about 25 minutes), and fifteen minutes after that, a hidden injection (into an IV line that extended a meter into another room) of either naloxone or saline is given. Group 7 (15 subjects, composed of “placebo responders”) gets open saline (presented as a painkiller) at the start, and a hidden injection of saline 15 minutes in. Group 8 (15 subjects, also “placebo responders”) gets saline (presented as a painkiller) at the start, and a hidden injection of naloxone 15 minutes later. Here is how they do:

For ten minutes after the hidden injection of saline or naltrexone, pain ratings continue to decrease. At 25 minutes, when the subjects in the two previously-discussed experiments would have long since tapped out, there starts to be a difference between the two groups: the saline group stays flat at a pain rating between 5 and 6, and the naltrexone group begins to rise back to the initial pain rating of 7. Here is Group 7 compared to no treatment non-placebo-responders, who report an increase in pain and tap out at 35 minutes:

Group 7 is also compared to Group 10, who gets a hidden injection of the drug proglumide, a drug thought to increase the effects of opiates:

They get a “placebo-enhancing” effect when .5 mg (but not .05 mg) of the drug is given as a hidden injection, when subjects who have already been hanging out in a tourniquet setup squeezing the squeezer for 25 minutes continue to squeeze and rate their pain as only between 4 and 5 after 45 minutes of more squeezing. The author does note:

After 45 min from the first injection, the experiment was discontinued even if pain ratings were still low. In fact, by considering these 45 min plus the waiting time to reach [a pain rating of] 7, the experiment lasted more than 1 h. After this time, subjects reported to be tired and usually asked to discontinue the experiment even if their pain was not rated as unbearable.

The next paper is Somatotopic Activation of Opioid Systems by Target-Directed
Expectations of Analgesia, by Benedetti et al. (1999). Here we get a break from the tourniquets and start doing subcutaneous injection of capsaicin into various body parts (hands and feet). Subjects were treated with a placebo cream that they were told was a powerful anesthetic, but only on one body part. “The specific expectation of analgesia on the left hand produced a placebo
effect only on the left hand and not on the other parts of the body,” the authors say. Further, “This highly specific placebo effect was completely blocked by naloxone.” The subjects gave lower pain ratings for the placebo-treated body part when injected with capsaicin, but not their other body parts similarly injected. However, subjects given naloxone did not give lower ratings for the placebo-cream-treated body part:

The authors also had a group of 20 subjects who received lidocaine cream rather than placebo, “to allow the double-blind design.” Unfortunately, the “natural history” control group is not reported to receive any kind of cream, so the “placebo effect” was that not just of suggestion of powerful pain relief, but whatever the cream itself may have contributed. Again, there was no effect of naloxone on the non-cream-treated natural history control group.

The fifth study, our first non-Benedetti study, is Gracely et al. (1983), Placebo and naloxone can alter post-surgical pain by separate mechanisms. Our subjects here are dental patients who just had their wisdom teeth out. Their instrument is the McGill Pain Questionnaire, which is a list of adjectives to describe various aspects of pain that translate into points (one point for flickering, five points for beating, two points for boring, three points for smarting, etc.) The maximum score is 78. The subjects were first given either 10 mg naloxone or placebo, and then treated with either fentanyl, placebo, or no treatment. They rated their pain on the McGill word scale 10 and 60 minutes before and after the treatment.

As for the fentanyl groups, their baseline pain was 17.44 (fentanyl only) and 16.88 (fentanyl but also got naloxone). Fentanyl alone reduced pain by 10.83 points at 10 minutes and 4.61 points at 60 minutes. Naloxone reversed the pain-relieving effects of fentanyl, decreasing pain by only .4 points at 10 minutes and increasing it by 2.8 points at 60 minutes.

The authors find effects for placebo treatment (getting a saline injection versus nothing), although smaller than for fentanyl. The “placebo effect” was also significant when the placebo recipients also got naloxone, but it was slightly smaller (filled-in circles):

Interestingly, the no-treatment group reported no change at 60 minutes after receiving a hidden placebo injection, but the no-treatment group reported much more pain after receiving a hidden naloxone injection. This is in contrast to the Bernedetti lab’s repeated finding of no effect of naloxone on natural history, a debate that will continue. The placebo group was polite as expected, experiencing a small and realistic placebo effect of perhaps 1-3 points on a 78-point scale. Naloxone seems to have caused a much greater difference for the no-treatment group than for the placebo group. They conclude, “naloxone hyperalgesia does not depend on placebo administration,” in contrast to the previous findings. They continue:

As this hyperalgesia is similar in the no-treatment and placebo groups, there is no evidence for an opioid component to placebo analgesia in this study. This finding suggests that naloxone hyperalgesia may involve the antagonism of endogenous opioid compounds released as a consequence of surgical stress. Although naloxone reverses stress analgesia, it has no effect on pain-free subjects…

“Stress analgesia” is the finding that if subjects are stressed out, such as by being forced to do math problems or public speaking, they will report lower pain ratings, as discussed in the main post.

So, we have our first non-Benedetti study, and the first failure to find naloxone extinguishing the placebo effect (especially compared to its effects on no treatment, much less fentanyl analgesia).

Sixth is another study from 1983, and another non-Benedetti study, Partial antagonism of placebo analgesia by naloxone, by Grevert et al. (1983). We are back to ischemic arm pain with tourniquets, and the authors use a 10 cm visual analog pain scale (which they do not say, but we can infer, is 0-100 points). During the tourniquet test, subjects rated their pain every minute for ten minutes. They showed up three times a week for three weeks, and underwent two pain procedures per day. The first procedure was always a control, with no injections or anything. In all sessions, subjects were hooked up to an IV apparatus, through which researchers could administer hidden injections. Only once per week, they received a placebo injection that they were told was a painkiller. (I’m not sure how the researchers justified setting up an IV line every session when no apparent injection occurred.) 40 minutes later, they received either naloxone or an inert substance in a hidden injection. On the other two “control” days, subjects did the regular control pain rating task and then received either a hidden injection of naloxone or the same inert substance, but no placebo. It took me way longer to figure out what was going on in this study than it took you to read this, but I apologize for how tedious it is. This is one of the most confusing studies I have ever read.

Subjects in the no-naloxone, hidden-sham-injection group demonstrated the largest decreases in pain ratings compared to their control sessions of about seven to ten points on a 100-point scale, versus the naloxone group, who seem to have a difference of between three and seven points compared to their control sessions. Here is placebo and inert hidden injection versus control sessions:

And here are placebo with naloxone versus their control sessions:

“The dependent variable is the change in pain score during the placebo session minus the change in pain score during the control session of the same week,” that is, (I think), not comparing pain scores, but comparing deltas of pain scores under different conditions. The authors assure us that the difference was statistically significant at p < .02. Therefore, we have our first (extremely confusing) non-Bernadetti result confirming that naloxone (slightly) reduces the (small) placebo effect.

Next we have Hersh et al. (1993), Narcotic receptor blockade and its effect on
the analgesic response to placebo and ibuprofen after oral surgery, another non-Bernadetti study. It is another study on patients experiencing pain after oral surgery. They tested ibuprofen, codeine, and placebo with and without naltrexone (another opioid antagonist), and collect an impressive array of measures of pain and drug efficacy.

Interestingly, despite collecting seven measures of drug efficacy, the authors could not even find a significant difference between subjects administered codeine (an opioid drug) with naltrexone (an opioid antagonist) and subjects given codeine and placebo. They also did not find any significant difference on any measure between placebo-naloxone and placebo-placebo, our second negative result for our third non-Benedetti study. The authors put this down to small sample size, despite their groups being of similar size to other studies reported. One of the only significant results that they could pull out, other than that ibuprofen was superior to the other treatments, was that naltrexone “significantly increased (p < .05) ibuprofen’s duration of action,” but I suspect that is just noise.

Next is Levine et al. (1978), The mechanism of placebo analgesia, the oldest study here and I believe the study that started it all. This is yet another study on oral surgery patients. They gave recovering patients injections at two hours and three hours after anesthesia, consisting of either morphine, placebo, or naltrexone for each injection, but only the results for placebo-placebo and placebo-naltrexone are reported. There results are summarized here:

They find a significant difference in pain ratings one hour after the second drug administration that is significant at p < .05 and looks suspiciously similar to the pre-existing difference between groups before any injection occurred. They further break down the subjects into “placebo responders” (n=11) and “placebo non-responders” (n=6) and find that naloxone only reduces the placebo effect in placebo responders.

Next we have another study by the same team, Levine and Gordon (1984), Influence of the method of drug administration on analgesic response. They are embracing technology in yet another study of impacted molar surgery patients, comparing open injections, hidden injections by a human, and hidden injections by a programmable robot syringe pump. Each method was used for two substances: naloxone and a placebo. Morphine was injected by the robot machine in two doses as a comparison.

In most studies, researchers first establish some kind of “placebo effect,” and then try to demonstrate that naloxone reduces it. Here, the researchers only give each patient one injection, with one baseline pain measure and one subsequent pain measure 50 minutes after injection. They infer a placebo effect by comparing the different injection method-substance interactions.

Pain seems to have increased for the group given only a machine injection of an inert substance (VEH for vehicle):

Although pain increases in all naloxone groups, the authors don’t expect that this is a result of naloxone increasing pain, but of natural history taking its course.

They infer a “placebo effect” from the observation that while the preprogrammed infusion pump injecting subjects with an inert substance increased their reports of pain greatly, an open injection of an inert substance decreased reported pain, and a hidden injection decreased it even more – a very strange placebo effect! Usually the difference between a hidden and open injection of an inert substance, with the open injection providing greater pain relief, is provided as evidence of a placebo effect. The authors conclude that

hidden infusion of vehicle may be accompanied by unintentional cues that elicit a placebo response, whereas machine infusion of vehicle does not provide such cues. Surprisingly, in post-experiment interviews, patients were unable to identify cues signalling when hidden infusions had occurred or what substance had been infused. Thus, subtle cues, of which a patient may not be consciously aware, can significantly influence therapeutic outcome.

Whereas I would conclude that it’s just noise in their 12-patient groups. But it’s lucky that they had the high-tech robot, otherwise it would be a negative result for placebo effects! Most of the studies rely on a hidden injection being undetectable, so to the extent that this study is to be believed, it casts doubt on many of the other results.

Now we have our last Benedetti paper, Pollo et al. 2003, Placebo analgesia and the heart. In the part of the study relevant to the meta-analysis, we are back to ischemic pain and tourniquets. They are using a pre-programmed infusion pump we just heard about for hidden injections. Each subject received either a hidden or an open injection. The groups relevant to us either got saline (hidden or open) or naloxone (hidden or open). Here are their results:

They find a small, significant placebo effect (the difference between hidden and open injections) in the saline group for both reported pain (p < .015) and heart rate (p < .035), but no significant difference (between hidden and open injections) for the naloxone group, although the naloxone group experienced a hilariously increased heart rate both before and after the pain trial. Although we know from Gelman and Stern (2006) that The difference between “significant” and “not significant” is not itself statistically significant, this insight came a few years late for this paper to benefit from it and perhaps offer further analysis. This is again using the method of inferring a placebo response from comparing hidden and open injections, rather than showing that naloxone reduces a previously-established placebo response.

Second-to-last is Posner and Burke (1985), The effects of naloxone on opiate and placebo analgesia in healthy volunteers, another non-Benedetti study. They compare the effects of naloxone on subjects going through a tourniquet/hand squeezer paradigm. Subjects in different groups received a capsule containing either the opioid drug dipipanone, a lactose placebo, codeine, or no capsule. They also received an infusion of either naloxone or saline, administered by a pump but not hidden from the subject (although whether it contained saline or naloxone was hidden).

The authors measured both subjective pain ratings and total time the subjects could stand the tourniquet. While they found that naltrexone did reduce the effects of dipipanone in terms of both pain ratings and scores, there were no significant differences between the ratings or times of subjects who received placebo and naltrexone, on the one hand, or placebo and saline, on the other. They did induce a placebo effect, as both were better than “no treatment” at p < .05, but naltrexone did not affect it. Another non-Benedetti negative result. (The relevant groups are P/S, placebo/saline, and P/N, placebo/naloxone, the triangles below. B/B and B/N are no treatment with and without naloxone; C is codeine.)

Finally, we have Roelofs et al. (2000), Expectations of analgesia do not affect spinal nociceptive R-III reflex activity: an experimental study into the mechanism of placebo-induced analgesia. As far as I can tell, this study is the only one to actually measure β-endorphins, which are supposed to be the whole point of this exercise. Sauro and Greenberg (2005) say that two studies did:

Two studies examined the effects of placebo administration on β-endorphin secretion. An effect size of 0.22 was calculated, but was insignificant (P = .36).

But after the review necessary to write this, I can’t figure out which other study they mean.

Roelofs et al. is the clearest and least confusing paper in the lot, specifying things that many papers leave up to the imagination (the procedure for randomization, the position subjects are to be seated in, how they store their data, etc.).

Roelofs et al. use electric shocks as their pain stimulus, with placebo groups (falsely) told they would receive fentanyl, and controls told they would receive either saline or naloxone. Each group completed three series of electric shocks and give pain ratings for each shock, one baseline, one after receiving saline (hyped as fentanyl or not), and one after a further injection if either naloxone or saline. They also measure β-endorphin levels from blood plasma samples taken four times, before and after each pain trial.

Surprisingly, despite (or because of?) their careful study design and practices, these authors don’t produce a significant placebo effect, reporting instead a very realistic -.5 points on a 100-point scale with a 95% CI of -5.8 to 4.8. β-endorphin levels were also not affected by the placebo billed as fentanyl, compared to honest placebo. And there was no effect of naloxone on the (nonexistent) placebo effect.

Study Arm Sizes

1. Amanzio M, Benedetti F. Neuropharmacological dissection of placebo analgesia: Expectation-activated opioid systems versus conditioning-activated subsystems. J Neurosci 1999; 19:484 –494.

• Control groups of 56 and 25 that are not relevant to the naloxone reduction of placebo effect.
• Study arms of 13 to 17 subjects each, as shown:

2. Amanzio M, Pollo A, Maggi G, Benedetti F. Response variability to analgesics: A role for non-specific activation of endogenous opioids. Pain 2001; 90:205 –215.

• Relevant study arms of 13-15 subjects as shown:

3. Benedetti F. The opposite effects of the opiate antagonist naloxone and the cholecystokinin antagonist proglumide on placebo analgesia. Pain 1996; 64:535 –543.

• Control groups of either 60 or 10 subjects that were not relevant to the naloxone placebo reduction. Study arms of 15 subjects in relevant arms.

4. Benedetti F, Arduino C, Amanzio M. Somatotopic activation of opioid systems by target-directed expectations of analgesia. J Neurosci 1999; 19:3639–3648.

• Control groups of 48 and 20 not relevant to the comparison. Relevant study arms of 24-29 subjects:

5. Gracely RH, Dubner R, Wolskee PJ, Deeter WR. Placebo and naloxone can alter post-surgical pain by separate mechanisms. Nature 1983; 306:264 –265.

• 89 subjects across 6 groups. I can’t find the group allocation; let me know if you can.

6. Grevert P, Albert LH, Goldstein A. Partial antagonism of placebo analgesia by naloxone. Pain 1983; 16:129 –143.

• One group of 16 compared to one group of 14. The only group not including Levine or Benedetti to confirm the finding.

7. Hersh EV, Ochs H, Quinn P, MacAfee K, Cooper SA. Narcotic receptor blockade and its effects on the analgesic response to placebo and ibuprofen after oral surgery. Oral Surg Oral Med Oral Pathol 1993; 75:539 –546.

• 11-16 subjects per relevant arm.

8. Levine JD, Gordon NC, Fields HL. The mechanism of placebo analgesia. Lancet 1978:654 –657.

• 17 and 23 subjects in the relevant comparison groups. Smaller subgroups are also analyzed.

9. Levine JD, Gordon NC. Influence of the method of drug administration on analgesic response. Nature 1984; 312:755 –756.

• 12 subjects in each relevant treatment group.

10. Pollo A, Vighetti S, Rainero I, Benedetti F. Placebo analgesia and the heart. Pain 2003; 102:125 –133.

• 14 or 15 subjects per relevant treatment group:

11. Posner J, Burke CA. The effects of naloxone on opiate and placebo analgesia in healthy volunteers. Psychopharmacol 1985; 87:468 – 472.

• 12 subjects per study with a crossover design.

12. Roelofs J, ter Riet G, Peters ML, Kessels AGH, Ruelen JPH, Menheere PPCA. Expectations of analgesia do not affect nociceptive R-III reflex activity: An experimental study into the mechanism of placebo-induced analgesia. Pain
2000; 89:75 –80.

• 13-15 subjects per relevant group.

Placebos for Depression

An interesting rejoinder to the Hróbjartsson and Gøtzsche meta-analyses is that of Wampold et al. (2005), in The Placebo Is Powerful: Estimating Placebo Effects in Medicine and Psychotherapy From Randomized Clinical Trials. They reanalyzed the studies from the original meta-analyses but reclassified them according to their own criteria, including whether psychology graduate student raters identified the studies as amenable to placebo treatment, and whether they thought the study design attenuated the placebo effect. Their conclusion was that in the subgroup of studies that they found conducive to placebo effects, the placebo effect “is robust and approaches the treatment effect.” They are particularly sanguine for the treatment of depression with placebos.

They seem to be correct that for accepted therapies for depression, the placebo effect (placebo minus no treatment) is almost as large as the treatment effect (treatment minus placebo). But this doesn’t seem to be a point in favor of a powerful placebo, but rather against the efficacy of accepted therapies. For example, Fernández‐López et al. (2022) attempted to estimate the placebo effect in the treatment of various mental disorders, including depression. They meta-analyzed studies that compared some kind of placebo arm (a pill placebo, “pseudo-meditation,” “bogus training,” “non-directive counselling,” “advice about how to be more organized,” etc.) with a no-treatment arm (wait list, no treatment, treatment as usual, etc.). In their random effects model for depression only, the overall placebo effect was SMD = .22 (95% CI of .04 to .39). While these are small studies, half of which occurred before the year 2000 and only three out of ten of which used a pill placebo, the small effect size seems plausible to me as a measure of politeness or response bias. It is likely smaller for pill placebos, as the authors did not find a significant effect for the subgroup of pill placebos, but this also included other conditions like schizophrenia, not just depression (and I would note that this is because pill placebos to me imply higher study quality, not because they are less “effective”). An effect size of .22 for placebo in depression is likely an overestimate, but I can’t find a better one.

Compare this to well-studied treatments for depression. Cuijpers et al. (2014) meta-analyze psychotherapy (talk therapy) compared to pill placebo on depression. Five of the ten studies they analyze use cognitive-behavioral therapy, a mind-cure therapy widely regarded as evidence-based and scientific. They find an effect size of .25 (95% CI .14-.36), and translate this to a simple effect size of 2.66 on the Hamilton Depression Rating Scale and 3.2 points on the Beck Depression Inventory compared to placebo. Furukawa et al’s 2017 meta-analysis of the five cognitive-behavioral therapy studies compared to pill placebo resulted in a SMD of .22 (95% CI .02-.42), identical to the placebo effect in the small meta-analysis above. As for the drugs, Stone et al. (2022) find an average drug-placebo difference of 1.75 points (95% confidence interval 1.63 to 1.86) on the Hamilton Depression Rating Scale in antidepressant trials submitted to the FDA.

Tangent: There are some other claims in the Stone et al. (2022) analysis that I will not get into detail about here, but one claim that has gotten a lot of attention is that Stone et al. find a latent distribution within the data that they label “Large,” having a 16-point mean improvement from baseline, and further find that 24.5% of drug-treated, vs only 9.6% of placebo-treated individuals fall within this latent distribution. This has been incorrectly interpreted as meaning that the drugs give a 15% greater chance for large improvement (e.g. Newsweek, “Antidepressants Work Better Than Sugar Pills Only 15 Percent of the Time“), but Stone et al.’s own supplementary figure shows that even this depressing mischaracterization is too optimistic:

Clearly, however things go for the unobservable-at-outset latent class that the authors identify, a class patients can’t know if they belong to when deciding on a treatment, the difference overall between drug and placebo at 16-points-or-greater appears to be much less than .15. End of tangent, back to the tiny effects of treatments for depression:

Turner et al. (2022) give a standardized effect size of .24 for newer antidepressants for trials submitted to the FDA and .29 for trials published in journals; for older antidepressants, the effect sizes are .31 for FDA trials and .41 for trials published in journals, possibly indicating a decreasing level of reporting bias. (Note that the drug studies represent 59 times as many patients as the talk therapy studies, so the fact that drug effects are well below the minimal clinically important difference of 3-6 points is probably more reliable than the clinically relevant difference for talk therapy on one self-report scale barely overlapping the clinically relevant difference of also 3-6 points, as estimated by Hengartner and Plöderl, 2022.)

In any case, the treatment effects are extremely modest, so modest as to not be noticeable by a typical patient or detectable by a typical clinician. While the certainty of evidence for the size of the placebo effect is very low, it is possible to conclude that, for conditions where treatment options are very poor, as meager as the placebo effect is, it might well approach the size of the treatment effect. This is hardly a point in favor of large placebo effects, even for self-reported criteria that can’t be distinguished from response bias. Patients in no-treatment groups in depression trials do very well, as it is an episodic disorder prone to regression to the mean, and patients in placebo groups do only slightly better.

Interestingly, although Hróbjartsson and Gøtzsche found a placebo effect for self-reported but not for observer-reported outcomes, in the case of antidepressant trials versus placebo, “placebo effects” seem to be larger when a researcher is doing the rating. In trials of depression treatments, both the placebo effect (Rief et al., 2009) and the treatment effect (Cuijpers et al., 2010) are larger for clinician-rated effects compared to self-reported effects. One interpretation of this is that the “placebo effect” in these trials is not so much from patients being polite and exaggerating their benefit, but from researchers exaggerating the change, either innocently or for the purpose of producing a larger apparent effect. Interestingly, a 2010 meta-analysis reported that this was also the case for Irritable Bowel Syndrome: higher placebo response rates for physician-reported than subject-reported outcomes. The negative result for outcomes measured by laboratory tests suggest this is exaggeration, rather than a genuine objective improvement.

by a literal banana

Nothing Works That Way

Back in 2014, Kevin Simler published a provocative essay on the nature of advertising called “Ads Don’t Work That Way.” I imagine most people reading this have read it, and if not they really should read it in its entirety, but I’ll summarize as a reminder. Some advertising works in really boring ways, basically reminding potential customers that the product exists. Simler gives the example of drain cleaner; I think this is also how most fast food advertising operates. But there is also a more psychological theory of advertising, which Simler calls “emotional inception,” in which advertisers create a Pavlovian association between their products and positive emotions or other desirable attributes. In the inception theory, our little brains are extremely malleable and vulnerable to such associations, and we can’t help but associate soda with happiness, sweetened fruit snacks with being a good parent, etc.

Simler does not think this is how advertising works. His theory, which he calls “cultural imprinting,” ascribes more rationality and less pathetic malleability to our interactions with advertisements. In this model, brands create cultural messages with their advertising dollars, so that buyers know what message they can expect to be sending when they purchase products that are consumed socially. One of his major examples is beer branding, and he predicts the Bud Light advertising debacle of 2023 with incredible accuracy with one bit flipped:

If I’m going to bring Corona to a party or backyard barbecue, I need to feel confident that the message I intend to send (based on my own understanding of Corona’s cultural image) is the message that will be received. Maybe I’m comfortable associating myself with a beach-vibes beer. But if I’m worried that everyone else has been watching different ads (“Corona: a beer for Christians”), then I’ll be a lot more skittish about my purchase.

For me, this was an introduction to a new way of thinking: perhaps those phenomena that popular media and official science explain with mysterious psychological effects on weak brains – automaticity – might actually be better explained by rational processes. The rationalist community centered on LessWrong, which was an important influence on my thinking, often focused on cognitive biases, taking the work of Daniel Kahneman and even priming studies seriously as evidence for the structures of human reasoning. To their credit, these associations do not seem to have been edited out of their corpus since the replication crisis in social sciences began to demolish the automaticity literature. An important motivation of the rationalist movement, as I saw it, was that we were all very irrational beings, and had to struggle to become more rational. My argument in this essay is that we are actually very rational, but managed to convince ourselves, for a variety of (perfectly rational) reasons using a variety of tactics, that we were helpless idiots.

Ego Depletion, or Thinking Fast and Slow, The Foundation of Priming

I take the term “automaticity” from the priming researcher John Bargh, famous for “proving” that simply solving word scrambles with elderly-related words like “wrinkle” or “Florida” caused undergraduates to walk as slowly as old people. (Fewer people know that he also “proved” that the same primes cause people to be more forgetful!) 

Let’s briefly look at what Daniel Kahneman had to say about priming research in his book Thinking Fast and Slow:

When I describe priming studies to audiences, the reaction is often disbelief. This is not a surprise: System 2 believes that it is in charge and that it knows the reasons for its choices. Questions are probably cropping up in your mind as well: How is it possible for such trivial manipulations of the context to have such large effects? Do these experiments demonstrate that we are completely at the mercy of whatever primes the environment provides at any moment? Of course not. The effects of the primes are robust but not necessarily large. Among a hundred voters, only a few whose initial preferences were uncertain will vote differently about a school issue if their precinct is located in a school rather than in a church—but a few percent could tip an election. 

The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you. If you had been exposed to a screen saver of floating dollar bills, you too would likely have picked up fewer pencils to help a clumsy stranger. You do not believe that these results apply to you because they correspond to nothing in your subjective experience. But your subjective experience consists largely of the story that your System 2 tells itself about what is going on. Priming phenomena arise in System 1, and you have no conscious access to them.

He was also a believer, at the time, of another important phenomenon, “ego depletion,” which, as I will explain, is the foundation for priming-style automaticity and in fact most of the phenomena that I label as examples of automaticity:

A series of surprising experiments by the psychologist Roy Baumeister and his colleagues has shown conclusively that all variants of voluntary effort—cognitive, emotional, or physical—draw at least partly on a shared pool of mental energy. Their experiments involve successive rather than simultaneous tasks. 

Baumeister’s group has repeatedly found that an effort of will or self-control is tiring; if you have had to force yourself to do something, you are less willing or less able to exert self-control when the next challenge comes around. The phenomenon has been named ego depletion. In a typical demonstration, participants who are instructed to stifle their emotional reaction to an emotionally charged film will later perform poorly on a test of physical stamina—how long they can maintain a strong grip on a dynamometer in spite of increasing discomfort. The emotional effort in the first phase of the experiment reduces the ability to withstand the pain of sustained muscle contraction, and ego-depleted people therefore succumb more quickly to the urge to quit. In another experiment, people are first depleted by a task in which they eat virtuous foods such as radishes and celery while resisting the temptation to indulge in chocolate and rich cookies. Later, these people will give up earlier than normal when faced with a difficult cognitive task.

Ego depletion is an embarrassing moment for science: hundreds of experiments “replicated” an essentially fake phenomenon, and it took the organized efforts of preregistered ManyLabs-style replication police to put it to rest. (I should mention that one of several large replication attempts led by a pro-ego-depletion researcher managed to find a small effect, but this needs to be weighed against the other large replication efforts, the most recent led by another pro-ego-depletion researcher, Kathleen Vohs, that failed to find any such effect, and against the general silliness of the project.)

But why did they fight so hard for ego depletion? It’s because ego depletion is the foundation upon which priming and other automaticity effects rest, and if it falls, they have nowhere to stand, as outlined by Bargh:

Tice and Baumeister concluded after their series of eight [ego depletion] experiments that because even minor acts of self-control, such as making a simple choice, use up this limited self-regulatory resource, such conscious acts of self-regulation can occur only rarely in the course of one’s day. Even as they were defending the importance of the conscious self for guiding behavior, Baumeister et al. (1998, p. 1252; also Baumeister & Sommer, 1997) concluded it plays a causal role only 5% or so of the time. 

Given one’s understandable desire to believe in free will and self-determination, it may be hard to bear that most of daily life is driven by automatic, nonconscious mental processes-but it appears impossible, from these findings, that conscious control could be up to the job. As Sherlock Holmes was fond of telling Dr. Watson, when one eliminates the impossible, whatever remains-however improbable-must be the truth.

For priming to exist as an important phenomenon, we must spend most of our time basically unconscious, or in “System 1” as Kahneman puts it, mere puppets of our environment, until some rare challenge forces us to wake up briefly to deal with it, so that we may go back to sleep.

Bargh and Kahneman are adamant that we can’t trust our experience, because our experience by necessity excludes the 95% of the time we spend as unconscious automatons, and that we must trust science instead. This is similar to the promotion of the “emotional inception” theory of advertising, which remains popular in marketing science, such as it is.

Here’s a bit of such science from Shiv, Carmon, and the legendary Dan Ariely, for flavor, which unites priming, “emotional inception” marketing, and the placebo effect, which will be the the subject of the next section:

In this experiment, participants first consumed SoBe Adrenaline Rush (a drink that claims to help increase mental acuity on its package) and then solved a series of puzzles.

Remember SoBe? The researchers told the subjects that they would be drinking this mind-improving beverage, and then had the gall to give them a form explaining that they were charging the subjects’ university accounts for the privilege of drinking it. Half of the subjects were charged the regular $1.89 price; half were told that they were only charged 89 cents, explaining it had been purchased with an institutional discount. What these experimenters “found,” was that if they had the subjects rate their “expectancies” for how much the SoBe would improve their mental functioning, thereby incepting such expectancies, the subjects who got it at a discount solved only 5.8 word jumbles, compared to 9.9 in the full price group. That is, simply drinking discount energy drink caused these poor subjects placebo brain damage to a fairly significant degree.

The example they give of a word jumble is “TUPPIL, the solution for which is PULPIT.” They allowed subjects 30 minutes to solve as many puzzles as possible; as the best group mean performance was about 10, they were taking about three minutes per word at the fastest. When I’ve looked up six-letter word jumble puzzles in the literature, they seem to allow about 20 seconds for subjects to solve them, but perhaps these were unusually hard puzzles. 

That is how priming is supposed to work: we are automatons going around in our sleep, and our performance on a simple puzzle can take a major hit simply by being informed that our drink was bought at a discount. We are infinitely vulnerable to our environment, to suggestion, to parlor tricks, that we can experience a major loss of intellectual ability, walking speed, memory, just by exposure to some infinitely subtle stimulus. 

From an evolutionary perspective, it seems like a bad design. You could be out there hunting with your elderly father, and suddenly you look at him and start walking slower and can’t remember what you were doing. You’re at your job and see an acronym BIB and start crawling around on the floor. You’re at the grocery store and you buy ice cream not because it is delicious and enjoyable but because the packaging primed warm emotional feelings and you subconsciously need to cool down. You find out that your prescription Adderall was bought with an insurance discount and suddenly lose 40% of your mental capacity. 

It’s not just that the relevant science is fabricated, or p-hacked, or uses meaningless measures or flexible measures, although the prevalence of such things does cast doubt on their evidentiary value. It’s that we should have been more skeptical from the start. The automaticity hypothesis is just as woo as spoon bending and precognition, and we should demand as extraordinary evidence for automaticity as we learned to demand regarding the others. See, e.g., Daryl Bem’s 2011 paper purporting to find evidence of clairvoyance, which may have played a role in instigating the replication crisis, and the history of the scientific investigation of spoon bending.

The spoon benders actually had to make a show of bending a spoon. Regarding automaticity, all we seem to ask is that someone wrote a paper claiming they bent a spoon. 

Pre-Post-Erous!

Most people take the placebo effect, as “demonstrated” in the SoBe study, for granted. We “know” that placebo pills heal people; maybe we even “know” that a placebo can still heal if it is openly labeled as such, or that more expensive placebos are more effective. If the placebo effect were not real, why would large medical trials of new drugs have to randomize subjects to a placebo condition? And why would they find big improvements in the placebo condition?

I will argue that we should put healing placebos into the “automaticity = woo” mental bucket. Placebos don’t work that way either. 

The first piece of the puzzle is how placebos actually function. Placebos have a perfectly valid job in randomized controlled trials: they are an attempt at mimicking the “noise,” or natural variation, from every aspect of the treatment process other than that believed to be efficacious, including time. Blease et al. explain the distinction: 

Before reviewing findings from OLP studies, it is crucial to clearly demarcate between two distinctive uses for the term placebo. First, is the usage of placebos in RCTs. Here the term is often understood to refer to a certain kind of ‘thing’ (eg, saline injections or sugar pills). Strictly speaking, this interpretation is incorrect: instead, placebos in RCTs ought to be conceived as methodological tools since their function is to duplicate the ‘noise’ associated with clinical trials including spontaneous remission, regression to the mean, Hawthorne effects and placebo effects. Properly understood, then, these types of placebos are deployed as controls that are specifically designed to evaluate the difference—if any—between a control group and a particular treatment under scrutiny. Ideally, in RCTs, controls should mimic the appearance and modality of the particular treatment or medical intervention under investigation. In contrast, placebos in clinical contexts are interventions that may be intentionally or unintentionally administered by practitioners either with the goal of placating patients and/or of eliciting placebo effects.

Many conditions are episodic or vary in severity, and starting from a bad time, the problem will often get better after a while on its own. When people talk about “regression to the mean” as an explanation for the placebo effect, this is what they mean. It’s mostly the “pre-post” comparison, as hinted in my title for this section. This effect can be enhanced by something called “eligibility creep” in dermatology: in an effort to include more subjects, researchers may exaggerate potential subjects’ condition at the outset of the study, so that an accurate measurement without exaggeration at the end will show a spurious improvement.

The second piece of the puzzle is that placebo effects compared to no treatment are small. Not just “small” in the meaningless sense of effect size, but too small to be noticeable or to make any clinical difference. For pain, a large meta-analysis found the mean placebo effect to be about 3.2 points on a 100-point scale, too small to matter. Similarly, the most recent meta-analysis on the placebo effect in depression found similar results, an effect size of .22, which is smaller than the effect of antidepressants over placebo, about .3, which translates into about 1.7 points on a 52-point scale and is too small to be clinically relevant. This is the case even though subjects in “no treatment” groups may have an incentive to exaggerate their symptoms in order to receive treatment, whereas those on placebo believe they are receiving treatment and have no such incentive. Still, effects are tiny, so tiny as to be meaningless in real life, and definitely tiny enough to turn out to be nothing at all with better methods.

The third piece of the puzzle is that we can’t trust studies that claim to find large placebo effects, for the same reason that we can’t trust priming studies. Here’s a figure from a paper by Waber, Shiv (from the SoBe paper), Carmon (same), and Dan Ariely, demonstrating a massive effect of more expensive placebos on pain from electric shock (both are placebos, the dark one is the normal price and the light one is discount):

We see effects in the “regular price” condition as high as 30 points on a 100-point scale, and often in the 20s, several times higher than the 3.2 point effect on pain we saw earlier, and certainly clinically significant. This level of pain relief would really matter. I was able to find a more recent study using similar methodology, though not labeling itself a replication, that found placebo effects of zero, one, and three points on a one hundred point scale in similar conditions, respectively. These seem much more realistic, and when combined with the meta-analysis results, indicates that the study that got a huge result is a major outlier. 

I do not think, contra Kahneman, that we “have no choice” to accept that such effects are real. Even without knowing about all the irregularities of this particular experiment (such as lack of IRB approval for shocking human subjects and deceiving them about pain medication), we should recognize how extraordinary these claims are, and demand appropriate evidence for them, not just one team’s word that such experiments even took place. 

Enter the Nudgelords

Nudge is just priming.

If there are stylized sculptures of waifish humanoids in a room, subjects will eat four fewer blueberries. If you sign at the top, you’re more honest than if you sign at the bottom. It’s priming, as a service.

Nudge studies aren’t real and don’t replicate. When they’re attempted in the real world, the effects are much smaller than in the academic studies, and the effective so-called nudges tend to share a curious feature: they operate on rationality rather than automaticity. For instance, one of the most effective “nudges” is apparently for the government to send clearly-worded letters explaining what they want people to do. That seems more like common sense and respect than a “nudge” to me, but I’m not a Nudge Professional. 

“Behavioral” anything (economics, finance, etc.) tends to reduce to automaticity explanations. 

I won’t spend much time on Nudge because it’s literally the same thing as priming – the same model of reality, the same experiments, often the same researchers, just using a different word. 

Cognitive Bias Parlor Tricks

I can’t go over every cognitive bias individually in this format, but I will give a basic pattern of how I think “cognitive biases” are produced as academic products:

First, an experiment or test is devised that people perform “poorly” on in some specific manner or dimension. It could be a list of choices of lotteries with different payoff structures (see e.g. this investigation of problems with a celebrated Kahneman mathematical model and associated experiments), or the famous demonstration of the “endowment effect” many of us experienced firsthand, in which we are given a unique little gift, like a mug, and then shamed for not being “rational” enough about its value when invited to trade it for somebody else’s gift. 

Second, this little experiment, and variants of it, are generalized to the entirety of human behavior. Perhaps a study on “conversation” involves strangers chatting in a laboratory about the answers to trivia questions, and the researchers find that this has no effect on accuracy. This may be generalized to a proposition like “conversations serve no purpose” in general, perhaps a “conversation bias” can be introduced, and people can feel smug for not liking meetings at work. 

Thus, we have scientific confirmation that a “bias” exists. This is often confusing, because, for example, if the “endowment effect” obtained under normal circumstances, markets would grind to a halt and not be able to function, because everyone would value what they already own more than anyone who didn’t own it. 

One solution to this kind of problem, both in cognitive bias and priming research, is to say that you found a “boundary condition” instead of that you failed to find the effect. “Boundary conditions” are a similar kind of cope to “mediator” or “subgroup” analysis, especially in small  studies not powered to detect them. 

Overall, I think rationality is a better starting assumption for human behavior, and we should demand a great deal of evidence for an important, widespread departure from rationality. 

Which of your favorite biases survived the replication crisis, and do they generalize?

Can You Catch Stupid?

“Social contagion” is the idea that behaviors spread through human groups like infections. Trivially, the spread of technologies like hybrid corn or mobile phones can be modeled like epidemics. The only “automaticity” aspect is the tendency to take the metaphor seriously, as if people were really affected by “social contagions” in the same unconscious, unfree way as by germs. 

Some phenomena that have been proposed to be socially contagious are suicide, obesity, quitting smoking, and (as a bit) acne, headaches, and height. The latter three are a bit in the sense that they use similar methodologies as the social contagion literature to demonstrate that homophily, the tendency for similar people to cluster in social groups, accounts for most, if not all, of the supposed “social contagion.” A more advanced method is to use time lags, as if time-lagged obesity weren’t as much a factor of homophily as snapshot-in-time obesity. 

However, researchers essentially never try to distinguish germ-type “contagion” from social learning. I think our starting hypothesis should be that behaviors that spread in the population arise from social learning, rather than from a mysterious unconscious process of thoughtless copying. Human copying is anything but thoughtless. Copying is an important form of creativity. Nonhuman animals can copy but a tiny fraction of our behaviors, even if we sometimes like to pretend they are capable of stealing our cool cottagecore aesthetics by dressing them up in sick outfits. 

People share and copy, but social contagion doesn’t work that way.

The Clockwork Universe With Clockwork People

An extended excerpt from the phenomenologist Gian-Carlo Rota, better known in mathematics for his work in combinatorics, on the fading myth of our time:

The theory of myths asserts that every civilization is ultimately characterized by the series of myths it believes in. These “working myths” are myths that that civilization is not aware of at all. They are not verbalized. The moment such a myth is out into words, it’s no longer something that people authentically believe in. It can now become a subject of discussion. So, as time goes by, a given myth that was universally believed in suddenly cracks and becomes doubted. At such a moment, the members of that civilization split into opposing camps: those who say “Yes, it’s so” and those who say “No, it’s nonsense.” And then the myth will fade, and finally it’s viewed as untenable except by a small group of people, who keep it as a superstitious belief.

….What I’m working towards is that not too long ago, there was a particular myth, universally believed, that in our time is being verbalized, which is the first stage towards its fading. A lot of phenomenology is a discussion and critique of that particular myth. 

What is that myth? It’s the myth of the clockwork, the myth of mechanism. It’s the idea that you can explain every phenomenon causally, by finding an underlying mechanism. It has been strongly believed in for a few hundred years. It had its heyday in the nineteenth century. It’s very simple: understand how the wheels work and you’ll understand everything. This went on and on, and that’s the myth that’s cracking in our time. I’m not in any way implying that mechanisms are bad, or that there are no mechanistic explanations. Certain phenomena can certainly be explained mechanistically. But there are other phenomena that do not have such an explanation. When you say this nowadays, tempers run high. Someone will point a finger at you and say, “You are an irrationalist! Either you believe in mechanism, or else you are irrationalist! Either a marble is red or it’s blue! Nothing in between!” But the phenomenological answer is that it is in no way irrational to deny a mechanistic explanation. We are not denying other modes of explanation which are logical and coherent, but just not mechanistic.

Those who will accuse us of irrationalism have an extremely narrow view of rationalism, a view that scientists have abandoned a long time ago. Ever since quantum mechanics came to be, and quantum mechanics is probably the greatest scientific idea of this century (n.b. the 20th -ed.), you can kiss goodbye to causal explanation. It’s very unfortunate that we still do not really understand quantum mechanics. Feynman used to stress that: quantum mechanics works, but there is something so mysterious about it, so completely different from anything we’ve thought about, that no one has succeeded in explaining it. But that works as our ally now. If someone tries to propose causal explanation as the scientific paradigm, we can say, “Phony baloney, science doesn’t work that way anymore! They gave that up decades ago.” Science is far more sophisticated. The strictly marble-like causal explanation is something that’s been thrown out of the window.

Science doesn’t work that way anymore, but few have gotten the message. Phenomenology proposes a different form of causality from the “marble-like” version, one based on conditions of possibility, a “Fundierung” relationship of things being founded on each other in the sense of allowing each other to come into being and be perceived as such. It is outside my scope to attempt an explanation at length, but this model of the world is not a woo model. It is a richer model, a more realistic model, and a model more in accord with careful observation of the world than the received marble-like model that has allowed us to accept so much silliness. 

The automaticity theories named here are holdovers from the fading myth of the clockwork universe, of clockwork people. The myth began to be named by the end of the 19th century (by William James at least, who also named the related Religion of Healthy-MIndedness that is still with us today). I hope by naming a more specific incarnation of the myth here that I can promote its thematization so that it can continue to fade. A science of ourselves cannot be established by dressing woo in lab coats, clipboards, and the mathematical ideas of an antique physics. 

I invite anyone to be the Lakatos to my Feyerabend, and present Here’s Why Automaticity Is Real Actually, as mine is an extreme case and does not pretend to be a measured, balanced examination of the subject. I would not recommend that anyone superstitious attempt this project, however, for obvious reasons: if priming were true, such an effort could prove lethal.

Here is the distinction as the authors put it, with citations removed:

Before reviewing findings from OLP studies, it is crucial to clearly demarcate between two distinctive uses for the term placebo. First, is the usage of placebos in RCTs. Here the term is often understood to refer to a certain kind of ‘thing’ (eg, saline injections or sugar pills). Strictly speaking, this interpretation is incorrect: instead, placebos in RCTs ought to be conceived as methodological tools since their function is to duplicate the ‘noise’ associated with clinical trials including spontaneous remission, regression to the mean, Hawthorne effects and placebo effects. Properly understood, then, these types of placebos are deployed as controls that are specifically designed to evaluate the difference—if any—between a control group and a particular treatment under scrutiny. Ideally, in RCTs, controls should mimic the appearance and modality of the particular treatment or medical intervention under investigation. In contrast, placebos in clinical contexts are interventions that may be intentionally or unintentionally administered by practitioners either with the goal of placating patients and/or of eliciting placebo effects.

Blease, C. R., Bernstein, M. H., & Locher, C. (2020). Open-label placebo clinical trials: is it the rationale, the interaction or the pill?. BMJ evidence-based medicine, 25(5), 159-165.

On the one hand, there is the use of placebos in randomized controlled trials, in which the point is to “duplicate the noise” that’s likely to exist in the treatment group. On the other hand, there are hypothesized “placebo effects” that may take the form of real healing, which is not at all the same.

For a specific example, just because antidepressant trials result in enormous placebo effects does not mean that depression responds to placebo in real life. The proper conclusion to the size of placebo effects in these trials is that the measurement of depression is extremely noisy, to put it in the most polite way.

While true “placebo effects” of the healing variety may exist, it’s worth engaging with these authors’ concerns over how that may be demonstrated, particularly in open-placebo design trials in which the hope is to pave the way toward ethical placebo treatment. The choice of control is particularly tricky; for example, as with antidepressant treatments, simply using “treatment as usual” or “wait list” as controls likely inflates apparent effects. True blinding requires a great deal of subtlety and effort in research design.

In summary: noise isn’t healing.

Now we can all pretend that we knew it all along and never mistook the one for the other!

This is not a troll. I am deadly serious and I think this is extremely important.

The sacred belief I am addressing here seems to have originated in the late twentieth century, and blossomed into ubiquity in the twenty-first century. It concerns “mental health,” and has multiple parts. First, it is the belief that mental illnesses are real diseases, just as serious as ordinary medical diseases, and no more the fault of the sufferer. Responses to trolls like “depression is a choice” bring this belief into visibility. Second, it is the belief that shame and stigma prevent people from seeking help for mental illness. Third, and the only part of the sacred belief system that concerns me here, is the belief that effective treatments for mental illnesses are available, and that once a person overcomes shame and stigma to seek help, the sufferer stands a good chance of getting meaningfully better.

Here, I will focus on one of the most common mental health problems, depression, known in current medical jargon as Major Depressive Disorder. Further, I will focus on the two gold-standard treatments for depression, generally considered to be the most effective treatments: antidepressant medication and cognitive behavioral therapy. I hope to eventually address treatments for two more conditions, schizophrenia and substance abuse disorder, but those will have to wait for future installments.

Medicine has always existed in some form, and there have always been people who claim to have been cured by the techniques of the day, going back at least as far as recorded history and almost certainly much further. But the evidentiary technique that is supposed to separate modern medicine from the misguided attempts at medicine of the human past is the double-blind placebo-controlled trial. Individual trials may lack statistical power and suffer from publication bias and other quality problems, however, so the true best evidence is probably the large meta-analysis of placebo-controlled trials. 

Meta-analyses raised concerns about the efficacy of antidepressant medications almost as soon as they began, but I will focus on a recent large meta-analysis, Cipriani et al., 2018, that received a great deal of attention. Cipriani et al. interpreted their findings thus: “All antidepressants were more efficacious than placebo in adults with major depressive disorder.” This seems like good news, but the bad news was how much more efficacious the drugs were. They report an effect size (standardized mean difference compared to placebo) of .30. This effect is considered “small” according to tradition and current guidelines, but how small is small? 

Most antidepressant trials use one or both of two measurement instruments: either the Hamilton Depression Rating Scale (HAM-D or HDRS), a 52-point rating scale applied by researchers or doctors to judge how depressed a person is, and the Beck Depression Inventory, a self-rating scale. (Note: the HAM-D may sometimes be referred to as the HAM-D-17 or HDRS-17 because it has 17 items, but it is a 52-point scale, as each item may score multiple points toward the total.) An effect size of .3 corresponds to about two points on the 52-point HAM-D scale, which, on the face of it, if you happen to read the instrument itself, is not much. This is not even taking into account that methodological issues and questionable research practices may account for most or even all of the apparent superiority over placebo. A difference smaller than seven points may not even be detectable by clinicians. Interestingly, a Cochrane review of Dance and Movement Therapy rejected the therapy as not having clinical significance because it only reduced depression scores by slightly over seven points above (psychological) placebo, which was less than 25% of baseline in the relevant studies. By this standard, since HAM-D scores in the antidepressant trials were generally in the mid-to-high 20s, at least 6 or 7 points would be necessary to achieve clinical significance. The upshot is that antidepressants “work” to a degree that is much too subtle for anyone to notice, which is bad enough, but also increases suspicions that they don’t work at all.

There are many branching paths of cope that have been explored for the seemingly devastating result that antidepressants are all pretty similar to each other and move a 52-point depression scale by two points at best compared to placebo. One is that the problem is the scale. Maybe using a self-report scale would better capture the healing effects of antidepressants? However, the Beck did even worse than the HAM-D in placebo-controlled trials, so that one can’t be the case. Or perhaps antidepressants work very well for some people, but just not for most people? That line of thinking was crushed too – the variability in scores for placebo was indistinguishable from the variability in scores in the treatment arms. (Note: a previous version of this study, by the same authors, found that there were significant differences, and was retracted because it was wrong. Since the authors originally came to the opposite result, suggesting some amount of researcher allegiance to the hypothesis, the negative result seems especially likely to be valid.) 

Another line of cope is the idea that big placebo effects represent real healing, and that antidepressants should continue to be prescribed for their placebo effect alone. Unfortunately, a great deal of the “placebo effect” found in these studies is probably a result of poor methodology. In the field of dermatology, this is known as “eligibility creep” – researchers inflate scores at baseline in order to qualify subjects, and then don’t inflate the scores at later points of analysis. There does seem to be evidence that this occurs in antidepressant trials. Furthermore, antidepressant medications cause significant side effects, so even if we believed that placebo healing is real healing rather than the result of research bias and questionable research practices, such drugs would not be appropriate to prescribe for this effect. 

Perhaps it shouldn’t be surprising that SSRIs in particular are not effective in treating depression, as every few years, someone points out that the popular serotonin hypothesis of depression is not substantiated by evidence. The most recent is a 2022 review, summarized here. The authors respond to common objections here.  I have no reason to believe that this will be any more effective than similar efforts going back to the 1990s to debunk the serotonin hypothesis; the myth of serotonin seems as sticky and ineradicable as the myth of antidepressant efficacy. 

What about Cognitive Behavioral Therapy? If antidepressants can’t be expected to produce clinically relevant relief from symptoms of depression, what about the most-touted “evidence-based” form of talk therapy? If you have read meta-analyses comparing CBT to “psychological placebo,” you may have been enormous effect sizes reported. A psychological placebo is something like treatment as usual, a waiting list, or some kind of vague “talking to a therapist,” not a real pill placebo. Even the aforementioned “Dance and Movement Therapy” achieved big effects sizes against psychological placebo that bordered on clinical significance. While I do not believe that placebo healing is real healing, the pill placebo control does seem to be something of a questionable research practice limiter. These authors give some suggestions why. I am not familiar with all possible methods of questionable research practices in therapy trials, but it does seem like it’s harder to cheat against a pill placebo compared to a psychological placebo.

When Cognitive Behavioral Therapy is up against pill placebo, the effect size, according to the most recent meta-analysis I could find, is a mere .22 when using the HAM-D. When using the self-report Beck Depression Inventory, the result is indistinguishable from zero and non-significant. As meager as the results for antidepressants are, the results for CBT are even worse. Researchers seem to think that subjects get a trivial amount better, but subjects themselves seem to feel no better compared to pill placebo.

You might imagine that there would be an outcry challenging this analysis, but I wasn’t able to find one. A typical write-up of this finding was the following:

CBT can benefit patients with severe depression, say researchers

…When compared with pill placebo, CBT led to greater symptom reduction on average by a standardized mean difference of -0.22 (95% confidence interval -0.42 to -0.02; P=-.03) on the Hamilton Rating Scale for Depression. The researchers said that this meant that the number needed to treat is 12 in typical cases of major depression, where the expected placebo response rates may be 30-50%. This would compare favorably, they said, with the number needed to treat (9) that can be expected in antidepressants, with an effect size of 0.31 over placebo.

The community seems to have just accepted this finding as meaning that CBT works, without apparently addressing the fact that the effect represents less than a two-point drop on a 52-point symptom scale, which is, as explained above, not clinically significant and probably not even clinically detectable. 

One defense of CBT might be that, unlike antidepressants, at least talk therapy cannot be harmful. I am not sure I believe this. Although CBT is not approved for the treatment of major depression in bananas, my experience with CBT was initially promising: the hope that my bad emotions were caused by bad thoughts, and that deconstructing the bad thoughts would limit the occurrence of bad emotions. However, what I learned through a few weeks of CBT, deconstructing every bad thought and bad emotion, is that the frequency and intensity of bad emotions is not affected at all by reasoning. If anything, the thoughts that co-occurred with bad emotions got even more ridiculous. Without CBT, it might not have been clear that I had no control over the occurrence of bad emotions. This might be regarded by some as a harm. Initially, I’d suspected that the apparent large effect sizes for CBT were a result of subjects answering surveys differently – e.g. experiencing a pure bad emotion rather than identifying it as “guilt” – but apparently my hypothesis was wrong: it was bad controls all the time, and I had been insufficiently cynical.

I am also a bit surprised, given the sacred beliefs mentioned at the beginning, by the fact that the existence of CBT is not considered insulting. It is hard for me to distinguish the methodology of CBT and other talk therapy methodologies from the aforementioned troll “depression is a choice.” If it’s really a disease, how would it make sense for it to be treatable by thinking correctly instead of thinking wrong? But apparently most humans do not make this connection and hence do not thereby feel insulted. 

An interesting rejoinder to the evidence that psychiatric treatments are not particularly effective is that very little medicine is actually effective. Harriet Hall, biting every bullet, titles her article “Most Patients Get No Benefit From Most Drugs.” Certainly, the problem of medicine generally not being effective is not limited to the field of psychiatry. But does that make it somehow excusable that the top treatments for depression cannot produce any clinically relevant effect? If I found out that all shoes tend to degrade into uselessness in two days, it wouldn’t make me feel a lot better to find out that hats also degrade into uselessness in two days. 

If antidepressants and CBT are the best treatments available, and have no clinically significant effect on symptoms of depression, what “help” is reasonably available? If the “stigma and shame” preventing people from seeking mental health treatment disappeared overnight, and everyone got treatment – and this seems to have largely happened, as antidepressants and CBT are as popular as they have ever been – it seems unlikely to make any difference in outcome. If the best the field has to offer is a glorified placebo, perhaps it has no help to offer at all. If this is true in many fields of medicine, then the problem is multiplied rather than solved.

As a banana who lives among humans, I am naturally interested in humans, and in the social sciences they use to study themselves. This essay is my current response to the Thiel question: “What important truth do very few people agree with you on?” And my answer is that surveys are bullshit.

In the abstract, I think a lot of people would agree with me that surveys are bullshit. What I don’t think is widely known is how much “knowledge” is based on survey evidence, and what poor evidence it makes in the contexts in which it is used. The nutrition study that claims that eating hot chili peppers makes you live longer is based on surveys. The twin study about the heritability of joining a gang or carrying a gun is based on surveys of young people. The economics study claiming that long commutes reduce happiness is based on surveys, as are all studies of happiness, like the one that claims that people without a college degree are much less happy than they were in the 1970s. The study that claims that pornography is a substitute for marriage is based on surveys. That criminology statistic about domestic violence or sexual assault or drug use or the association of crime with personality factors is almost certainly based on surveys. (Violent crime studies and statistics are particularly likely to be based on extremely cursed instruments, especially the Conflict Tactics Scale, the Sexual Experiences Survey, and their descendants.) Medical studies of pain and fatigue rely on surveys. Almost every study of a psychiatric condition is based on surveys, even if an expert interviewer is taking the survey on the subject’s behalf (e.g. the Hamilton Depression Rating Scale). Many studies that purport to be about suicide are actually based on surveys of suicidal thoughts or behaviors. In the field of political science, election polls and elections themselves are surveys. 

What I mean by “surveys” is standard written (or spoken) instruments, composed mostly of language, that are administered to subjects, who give responses, and whose responses are treated as quantitative information, which may then be subjected to statistical analysis. It is not the case that knowledge can never be obtained in this manner. But the idea that there exists some survey, and some survey conditions, that might plausibly produce the knowledge claimed, tends to lead to a mental process of filling in the blanks, of giving the benefit of the doubt to surveys in the ordinary case. But, I think, the ordinary survey, in its ordinary conditions, is of no evidentiary value for any important claim. Just because there exist rare conditions where survey responses tightly map to some condition measurable in other ways does not mean that the vast majority of surveys have any value. 

Survey evidence seems to be a new phenomenon. Robert Groves (2011) argues that it is a 20th century phenomenon, arising in the 1930s, achieving a golden age from the 1960s to the 1990s, and then falling off in prestige and reliability after that. 

Why is it important that surveys are new? I think it is important to remember that there is no ancestral practice equivalent to surveys. That is to say, there is no ancient human practice or context in which people anonymously tell the pure, innocent truth with language, in response to questioning, with no thought for the motives of the questioner or the effect of their answers. However, in the new, wholly invented ethnomethod of [doing a survey], it is imagined that subjects do tell the innocent truth, comprehending the underlying sense of the question but not answering with any motive or particularity of context. The anonymity of survey takers is given as proof that they feel free to tell the truth, rather than being perceived as a bar to asking them what they might have meant by their responses.

To get at the philosophical weirdness of the survey, it is necessary to dissect the phenomenon of survey-taking in detail. First I will consider the legal perspective, since that is an ancient domain of getting at truth through language in particular contexts. 

Surveys and Legal Evidence

The vast majority of evidence in the legal context is and has always been testimonial. That is, a witness testifies in language to communicate some fact to the judge or jury, and the judge or jury then decides how much to believe it. This is true even for modern DNA evidence: the expert witness testifies about the alleged meaning of the laboratory findings, even if documents (also testimony of the writer or preparer) are also given to the trier of fact to examine. Good evidence and garbage evidence are both usually testimony. 

In the English common law tradition, the biggest rule about testimonial evidence is hearsay. To put it colloquially, the hearsay rule says that testimony has to come from the horse’s mouth. If you saw someone run a red light and crash into a fire truck, you could undergo the ritual of being put under oath (agreeing that negative legal consequences could befall you for untrue speech), and testify about what you saw in court. Then you would be subject to the further ritual of cross-examination, in which all the aspects of your testimony could be questioned: your vision, whether you had your glasses on, where you were positioned, what specifically you mean by “crashed,” your relationship with the driver, and perhaps even whether you had a past conviction for forgery, which might make you seem like a liar in general. Those responsible for deciding whether to believe your testimony would have the chance to look at your face and mannerisms while testifying, to look at your clothes and hair and grooming, to see whether your eyes are bloodshot, in order to judge your responses to questioning. This may seem superficial and unfortunate, but in conversation we make these judgments all the time. And a witness may lose credibility for appearing too slick as often as for appearing too tattered, as in ordinary life.

However, if you wanted to testify about what you heard someone else say, someone who isn’t present for ritual oath-taking and questioning, that would be hearsay, admissible only under a list of exceptions. The exceptions to the hearsay rule are generally contexts in which language evidence is considered particularly likely to be accurate and truthful, such as a record kept in the ordinary course of business, or an emotional shout just after the crash (the idea being that you wouldn’t have time to think up a lie). 

Survey evidence, then, is plainly hearsay, since when we hear claims based on survey evidence, we get no opportunity to judge the credibility of the statements as we might in conversation (much less under oath). However, survey evidence is often admissible in legal proceedings, particularly under the much-abused “state of mind” exception. But I think the more common reasoning underlying admission of survey evidence is as stated by a legal scholar at the beginning of the golden age of surveys (Zeisel 1959): “[S]ince surveys provide the best, if not the only, evidence on certain issues, and since expert knowledge in the field has advanced sufficiently to protect the trier of the facts from error, the law may well lower its heavy guard” (bolded emphasis mine). In other words, survey evidence is admissible because there’s no other way to get at the underlying facts. Consider trademark confusion: how would one measure whether consumers confuse one mark with another except by asking them in some clever way? The phenomenon of confusion is hidden in the minds of consumers, and can’t be measured with calipers or rulers. 

Even when surveys are the only way to get at some particular knowledge, they may be done well or poorly. Zeisel (1959), citing Coca-Cola v. Nehi Corp., 27 Del. Ch. 318, 326, 36 A.2d 156 (1944), says:

Other aspects of an interview can also become grounds for criticism. Word association tests given to students in a classroom were rejected because their reactions were “bound to differ from that of the buyer in the market place when confronted with the.., beverage …. ” As another court remarked, “the issue is not whether the goods would be confused by a casual observer, but [rather] .. .by a prospective purchaser at the time he considered making the purchase. If the interviewee is not in a buying mood but is just in a friendly mood answering a pollster, his degree of attention is quite different.”

That is to say, even though a survey might be the only way to judge the phenomenon of confusion, a college classroom was judged to be sufficiently different from shopping in a store (e.g.) to render the survey meaningless. I find this standard touchingly exacting compared to the present lax standard for taking survey evidence seriously. The present standard seems to be that the more math you do to survey data, the more reliable it is.

The meaning of my title is from a joke told at the end of Annie Hall:

I thought of that old joke—you know, this guy goes to a psychiatrist and say doc, my brother’s crazy! He thinks he’s a chicken. And the doc says, why don’t you turn him in? Then the guy says, I would, but I need the eggs. I guess that’s pretty much now how I feel about relationships. They’re totally crazy, irrational, and absurd, but I guess we keep going through it because most of us need the eggs.

Surveys are perhaps the only way to get certain information, information about the most important and pressing phenomena, about happiness and suffering in all its forms. These are eggs that most of us need. So even though surveys are bullshit, they are not “turned in” like the unfortunate brother in Woody Allen’s joke, but embraced in a plausibility structure whose maintenance is widespread and in which we are all complicit.

The Phenomenon of the Survey

To understand what’s wrong with surveys, we must alternate between critically examining survey instruments themselves, and inferring what we can about the context in which surveys are taken.

I want to go into some detail about the assumptions that the meaningfulness of surveys rest upon. When we say that some conclusion “rests upon” assumptions, the metaphor might be one of a house resting on a foundation, where if the foundation is not sound, a collapse may occur. But in this case it is more like the parts of a machine, which all must function. In the case of a steam engine, the water container must be sound and hold pressure without leaking, the piston must pop up and down, the wheel must spin freely, etc. If any part of the machine does not function, the machine does not function.

But in the case of surveys, even if all assumptions fail, if all the pieces of the machine fail to function, data is still produced. There is no collapse or apparent failure of the machinery. But the data produced are meaningless—perhaps unbeknownst to the audience, or even to the investigators. What follows is my attempt to identify the moving parts of survey meaningfulness, with some attention to how they interact. Keep in mind that all of these are based on an underlying assumption that there is no outright fraud—that data are gathered in the way stated, and not made up or altered, either by the researchers or by any of their subcontractors or employees. 

Given innocent scientists, the first of these interacting pieces is attention. In order for the survey to be meaningful, the subjects must be paying attention to the items. This is the first indication that the relationship between survey giver and survey taker is an adversarial one. The vast majority of surveys are conducted online, often by respondents who are paid for their time. (Just to give you an idea, I happened to see figures in recent publications of fifty cents for a ten-minute task and $1.65 for fifteen minutes.) A best practices document (Peifer & Garrett, 2014) identifies the practice of “satisficing” — when survey respondents exert what they perceive to be the minimum of cognitive effort that will not result in a rejected contribution. The survey taker wishes to complete as many tasks as possible, and cannot be supposed to have any motivation to bring his full self to the task. There are several strategies to detect careless responses, such as including “trap questions” and asking the same question in multiple ways, on the assumption that the change in wording will not change the meaning in any way to a sincere survey taker. However, it is likely that savvy survey takers quickly detect these strategies and find ways to minimize cognitive effort despite their presence. 

The second piece might be termed sincerity. It is this, I think, that is meant by the “lizardman constant” – a (facetiously) constant percentage of people who, asked if the world is run by lizardmen, will respond in the affirmative. In addition to a money motivation, survey takers may be motivated by having fun with the survey by purposely reporting wrong or absurd answers. To some degree, this is on a continuum with attention: answering a survey absurdly may be more fun, and hence require less cognitive effort, than answering it sincerely. The survey that found that gay teens are much more likely to be pregnant, or to get somebody pregnant, most likely reflects this phenomenon. Especially since surveys are generally anonymous, with the only identifying information supplied by the survey taker, it is generally impossible to check the degree of sincerity of survey respondents. Compare this to your personal life, in which you have a good idea about which of your friends is trolling at any given time, and who is unlikely to troll.

A related adversarial motivation is making a point. In the normal course of conversation, in ordinary language use, one forms opinions about why the speaker is saying what she is saying, and prepares a reply based as much on this as on the words actually said. In surveys, survey takers may form an opinion about the hypothesis the instrument is investigating, and conform his answers to what he thinks is the right answer. It’s a bit subtle, but it’s easy to see in the communicative form of twitter polls. When you see a poll, and your true answer doesn’t make your point as well as another answer, do you answer truthfully or try to make a point? What do you think all the other survey respondents are doing? This is not cheating except in a vary narrow sense. This is ordinary language use—making guesses about the reasons underlying a communication, and communicating back with that information in mind. It’s the survey form that’s artificial, offered as if it can preclude this kind of communication. And even when a survey manages to hide its true hypothesis, survey takers still may be guessing other hypotheses, and responding based on factors other than their own innocent truths. 

So we have attention, sincerity, and motivation as moving pieces, although they overlap messily with each other. The next piece can be summed up with the word comprehension, but this simple word hides a complex phenomenon that itself has many moving pieces. 

Here are a couple of questions from a recent study. These purport to measure “Paranoid ideation” (quantified on a nine-point Likert scale, and yes this is the whole instrument):

Every day, our society becomes more lawless and bestial, a person’s chances of being robbed, assaulted and even murdered go up and up. 

Although it may appear that things are constantly getting more dangerous and chaotic, it really isn’t so. Every era has its problems, and a person’s chances of living a safe, untroubled life are better today than ever before (reverse coded).

If you’re reading this, you probably slid right over the words “bestial,” “chaotic,” and “era.” But these are rather difficult words, and not everyone reads essays by bananas with words like “epistemology” and “indexicality” in them. You probably also slid right over the fact that the first item is not a grammatical sentence, but two sentences jammed together with a comma splice. Difficult words and baffling grammar are possible barriers to comprehension. But it is more complex than merely remembering to use simple language. Apparently simple words often mask complex structure and associations.

Consider another survey question, from the General Social Survey:

Taken all together, how would you say things are these days—would you say that you are very happy, pretty happy, or not too happy?

No one can accuse this of using big words (although my friend points out that “not too happy,” despite its colloquial meaning, is exactly as happy as you’d want to be). But in its simplicity, it exemplifies the complexity of the phenomenon of comprehension.

Consider what comprehension means here. It presumes first that the authors of the survey have encoded a meaning in the words, a meaning that the words will convey to the survey takers. More importantly, it presumes that this corresponds to “the real meaning” of the words—a meaning shared by the audience of the survey’s claims. What would the “real meaning” be in this very simple case? How are things these days? Are you very happy, pretty happy, or not too happy? What informs your choice? Would you have answered the same a month or a year ago? Fifteen minutes ago? How does your “pretty happy” compare with another person’s “pretty happy”? Happy compared to what? How would you predict that your family members would answer? Do they put a good face on things, or do they enjoy complaining? Would their answers correspond to how happy you think they really are? What about people from cultures you’re not familiar with? This is a three-point scale. Would you be able to notice a quarter of a point difference? What would that mean?

What underlying construct can we presume that the answer to this question gets at? If you ask enough people, will an underlying construct emerge where none existed before due to magic of the wisdom of crowds? This seems to me the mindset of divination. Divination (such as reading animal entrails) may actually be the ancient precursor to surveys that I was complaining didn’t exist in an earlier section. Divination practices are probably also a “most of us need the eggs” situation. National vibe checks on the General Social Survey might serve a similar purpose. Like traditional divination practices, it is guaranteed to produce an answer.

Comprehension is difficult enough in actual conversation, when mutual comprehension is a shared goal. Often people think they are talking about the same thing, and then find out that they meant two completely different things. A failure of comprehension can be discovered and repaired in conversation, can even be repaired as the reading of a text progresses, but it cannot be repaired in survey-taking. Data will be produced, whether they reflect the comprehension of a shared reality or not.

Here is another survey instrument:

It is not clear to me how much the content of these rather vague items reflects some underlying construct, existing in the world, of “conspiracy mentality,” versus other constructs. (I wonder if political historians score particularly high on the CMQ regardless of their personal tendency to conspiratorial thinking.) But a particularly interesting aspect is the stacking of abstractions. For instance, question 2 uses the word “usually,” a word with a probabilistic meaning, and question 4 uses the word “often.” Stacked on top of these estimates is the one hundred point rating scale, which helpfully supplies word meanings for the percentages. Other surveys have been conducted as to how people interpret probability words on a scale of percent likelihood and get varying spreads; it is unclear whether most survey respondents would interpret “somewhat unlikely” as “40%.” But survey respondents must complete the very difficult task of estimating the likelihood of whether something is true “often” or “usually.” Is it somewhat unlikely that something is usually the case?

What sense do the subjects make of the survey items? We cannot know. All we can know is the data they produce, regardless of whether any sense has been made. 

Assuming that survey subjects are paying attention and being sincere (which we have no reason to assume), and assuming that they comprehend the true sense of each survey item as it corresponds to some part of reality (again dubious), they must also be accurate in their judgments and reporting for the data to be meaningful. The possibility of accuracy rests on there being some underlying reality against which to measure, which is debatable in the above cases. But for things like nutrition studies, in which food consumption is measured by a survey, there is some underlying reality (the food eaten), and respondents must be accurate in their recall and complete in their reporting for these surveys to be meaningful. If you have ever tried to keep track of your food intake, and if you can imagine people at all levels of competence and conscientiousness attempting the same task, you have some idea of the degree to which nutrition study responses correspond to reality. I searched for studies telling you that red meat is bad for you, and the first one I encountered was using a survey that asked participants to recall their diet over the past twelve months. The authors of that study used the portion sizes and frequencies that people could recall over the past year to calculate how many grams of meat they ate per day. 

Of course, this is not the only problem with nutrition surveys. The more important issue is that answers to food survey questions are treated as quantitative data, and compared to various outcomes (many of which are also determined by surveys), with the assumption that differences in reported food consumption are causally related to health outcomes, while assiduously avoiding to the extent possible any consideration of factors that might confound the purported causal association. But it is important to keep in mind that even though there are many problems with nutritional analyses, they are generally based on survey data at their core.

I have presented the assumptions for survey meaningfulness in the order that seems most natural for understanding them, and finally come to what I regard as the most important factor, alluded to but not named in italics above. From comprehension and accuracy we can deduce the need for the existence of an underlying phenomenon that is being measured. In the case of nutrition surveys, there is a phenomenon—a fact of the matter—-that could be gotten at by, say, constant surveillance. People really do eat certain foods in certain quantities, whether or not it has much effect on their health or mortality. But for the really important issues, things like happiness and depression and political beliefs, there may really be no phenomenon underlying the attempt at measurement. This is not to say that happiness is not real, in the sense that it is subjectively felt that things are going well or poorly. Emotion and feeling and belief are real, as we can all see by experiencing them from time to time. We may even communicate about them in conversation, and use the communication and all the context surrounding it to judge the emotions and beliefs of others. My claim is that there is no basis for believing that a shared phenomenon underlies the use of words like these across contexts, especially the anti-context of survey taking. 

Biofluorescence is a hot phenomenon as I write. Museums and zoos and wildlife enthusiasts are shining ultraviolet light on their preserved pelts and living animals in the dark, and finding that such things as flying squirrels and rats and platypus and Tasmanian devils, not to mention frogs and spiders, absorb invisible (to human eyes) UV light and re-emit it in the visible wavelength. The fluorescence often forms attractive (to my eyes) patterns on the animals. These patterns were always there, in some sense, whether they are typically observed by some observer (a mate or predator, perhaps) or not. But now humans can see them, because they have a machine (darkness plus UV light source plus eyes and/or visible-light-sensitive camera) through which they can look. 

I like the image of the science machine as a dark room with an electric hum, illuminating colorful crystalline patterns that exist but were previously invisible to the onlookers, who peer through a dark screen at the radiant geometries on display. 

Now consider the happiness survey. There is a phenomenon that we can’t directly observe: how other people feel. The survey shines its light, in the form of questions in language, and there is a response, and there are any number of patterns to be seen in the data obtained. Does this mean that the patterns in the data are illuminating a pattern that exists separately in the world, like bioluminescent fur? Is there any such pattern at all to be illuminated? Or are they, perhaps, patterns of language use (happiness-survey-talk by real, rather than generic, people) rather than clues to the structure of the referents of language (happiness itself)? 

It is one thing to insist that words mean things while arguing pedantically about some dubious word usage, but it would be exhibiting a high degree of temerity to go so far as to believe that words mean things, much less to rely on this belief in making deductions.

Here is, I think, the most fundamental reason that surveys are of no evidentiary value: the phenomenon that the survey purports to measure either has no existence independent of the survey, or the words used in the survey instrument do not correspond to the real underlying phenomenon they purport to measure. No amount of attention and accuracy and innocence of motivation can make up for that. And I do not think the problem will be solved by improving the survey instruments, any more than priming studies will be rendered meaningful by better primes.

Peifer, J., & Garrett, K. (2014). Best practices for working with opt-in online panels. Ohio State University: Columbus.

Groves, R. M. (2011). Three eras of survey research. Public opinion quarterly, 75(5), 861-871.

Zeisel, H. (1959). Uniqueness of survey evidence. Cornell LQ, 45, 322.

by a literal banana

I have been trying to understand the “lexical hypothesis” of personality, and its modern descendant, the Five Factor Model of personality, for several months. In that time, I have said some provocative things about the Big Five, and even some unkind things that I admit were unbecoming to a banana. Here, I wish to situate the Five Factor Model in the context of its historical development and modern use, and to demonstrate to the reader the surprising accomplishment that it represents for the field of psychology.

In personality research, the “lexical hypothesis” refers to a hypothesis attributed to Francis Galton (1884). Galton supposed that each human language would reflect important realities of human character within that language and culture. In particular, he noted that the words used to evaluate character and personality are very numerous (he estimated over a thousand, using a thesaurus), and often overlap in meaning.

But Galton immediately left his thesaurus behind, readily admitting of the impossibility of defining any aspect of character. Rather, he turned to experimental means of testing the character in various ways, and insisted that no particular map or model of personality is needed to start from.

Nowhere in his essay does Galton propose surveys as a means for studying character. He would probably regard such methods as unscientific, as indicated in his final paragraph:

[C]haracter ought to be measured by carefully recorded acts, representative of the usual conduct. An ordinary generalisation is nothing more than a muddle of vague memories of inexact observations. It is an easy vice to generalise. We want lists of facts, every one of which may be separately verified, valued and revalued, and the whole accurately summed. It is the statistics of each man’s conduct in small every-day affairs, that will probably be found to give the simplest and most precise measure of his character.

The methods that Galton proposed are exclusively non-linguistic. For instance, he commented that observing children involved in play quickly gives one an idea of each child’s emotional expression. Galton’s proposed methods prefigure both hidden camera prank shows and Goffman’s “breaching experiments:”

I will not attempt to describe particular games of children or of others, nor to suggest experiments, more or less comic, that might be secretly made to elicit the manifestations we seek, as many such will occur to ingenious persons. They exist in abundance, and I feel sure that if two or three experimenters were to act zealously and judiciously together as secret accomplices, they would soon collect abundant statistics of conduct. They would gradually simplify their test conditions and extend their scope, learning to probe character more quickly and from more of its sides.

Other methods Galton expressed enthusiasm for include heart rate measurement (he wore a home-brew heart-rate-measuring apparatus while he delivered the lecture that makes up the text) and methods discoverable from personal context (giving an example from Benjamin Franklin, of a man with one attractive and one deformed leg, who kept track of which leg his interlocutors paid attention to, as a gauge of their optimism or pessimism). Galton would be surprised, I think, to find that the most promising and scientific theory of personality in the twenty-first century is premised entirely on survey responses as its “facts.”

Early in the study of personality, there was a major shift of meaning in the lexical hypothesis. At first, the thesaurus and the word list were its tools of study (e.g., Allport & Odbert, 1936); the idea was to find common factors of meaning in the words themselves. Of course, there is no particularly scientific way to decide how much the word “annoying” is the same as “obnoxious,” or how much either is the same as “low-status.” The major shift was to begin to measure the correlations of an entirely different construct: the correlations of the words when used to describe a particular person. That is, rather than trying to measure the underlying meaning of words, researchers began to measure the degree to which different words were applied to the same person. “Sameness” and “correlation” were no longer distinguishable concepts for the methods.

Initially, lists of adjectives, and eventually, short survey questions, were administered to subjects, who described either a person they knew or themselves. When the responses were subjected to factor analysis—a mathematical analysis to reveal the structure of correlations between responses—a varying number of factors emerged, depending on the methods and the researchers and the questions and the subjects, and these factors were given varying names. Since the early 1990s, the Five Factor Model has been dominant, although the names of the factors vary somewhat even today. The acronym OCEAN is used for the traits: Openness to experience (sometimes called “intellect” or “imagination” or “open-mindedness”), Conscientiousness, Extraversion (sometimes called “surgency”), Agreeableness, and Neuroticism (sometimes called “negative emotionality” or “emotional stability” reversed). 

Today, the five traits are measured with various survey instruments, with five questions on the shortest version (one for each aspect) and sixty questions on a common long-form version (that used by Soto, 2019). Survey instruments are validated in a number of ways: how much their responses correlate between testings (test-retest reliability, with astrological sign as the gold standard), how much different raters agree using the criteria (inter-rater reliability), and a nebulous concept of construct validity, which sometimes includes scientific gestures designed to ensure that the instrument measures what it purports to measure. Many papers present elaborate numerical artifacts of validation, and I have found that some characterize the validity of their instruments as “good” without providing an indication of what would be “not good enough.” From a brief review of dozens of validated instruments in social psychology, it seems to me that it is relatively easy to “validate” meaningless instruments. As long as the mathematical bona fides are present, the construct need not be meaningful in other ways. (The reader who is rightly suspicious of my broad and unsourced claims may wish to search Google Scholar with variations of “scale,” “inventory,” and “survey instrument,” and examine the results critically. The naming of factors is often a particularly interesting step.)

The strong claim made by advocates of the Five Factor Model is that any set of questions describing a human being, administered to subjects and the responses subjected to factor analysis, will reveal the same five factors (paraphrasing Jordan Peterson in this video, around 11:10-16:40). This strong claim, though dubious in a number of respects, is a major part of the basis for the scientific legitimacy of the Big Five. It is interesting to see which aspects of the strong claim are admitted to be false by advocates of the Big Five, and how much is excused on the grounds that at least it’s something. The Five Factor Model is not perfect, advocates grant, but it is better than nothing. It is not clear how they measure “better than nothing;” this is a potentially interesting hypothesis in need of precisification, perhaps.

The Big Five exist as a special, scientifically validated property of language and survey methods, and that is one basis for their legitimacy. The other basis for the legitimacy of the Five Factor Model is its replicable correlation with consequential life outcomes. We know that the Big Five are not merely phantoms that fall out of a certain analysis of a certain use of language in WEIRD college students, because these traits are reliably correlated with things we care about. 

One of the most interesting features of the Big Five is the nature of its scientific evidence. Observe what is held out as a “replication” of the theory, and you will discover the theory’s true nature. The most impressive aspect of the ongoing accomplishment of the Five Factor Model is the degree to which it deflects curiosity about its underlying meaning with rituals of scientific validation, regardless of the rituals’ appropriateness in context. Since “replication” is the scientific ritual most recently shown to detect poor science in psychology, being shown to reliably “replicate” is a huge boost to the credibility of a theory. 

The interesting thing about the Five Factor Model is what it gets away with, in terms of being considered a theory, even though it is not causal, and makes no predictions. What counts as a “replication” of the Five Factor Model, as in Soto (2019), is the following: a correlation is found between one or more factors of the Five Factor Model and some other construct, and that correlation is found again in another sample, regardless of the size of the correlation. In almost all cases, and in 100% of Soto (2019)’s measures, the construct compared to a Big Five factor is derived from an online survey instrument.

What counts as a “consequential life outcome” is also fascinating. In most cases, the life outcome constructs are vague abstractions measured with survey instruments, much like the Big Five themselves. For instance, the life outcome “Inspiration” is measured with the Inspiration Scale, which asks the subject in four ways how often and how deeply inspired they are. Amazingly, this scale correlates a little bit with Extraversion and with Open-mindedness. Do these personality traits “predict” the life outcome of inspiration? Is “Inspiration” as instrumentalized here meaningfully different from the Big Five constructs, such that this correlation is meaningful? 

Compare the items for the construct “Inspiration” with the items for Extraversion and Open-mindedness used in Soto (2019):

Inspiration Scale Items

• I experience inspiration.
• Something I encounter or experience inspires me.
• I am inspired to do something.
• I feel inspired.

Extraversion+ Items

• 1.   Is outgoing, sociable. 
• 6.   Has an assertive personality.
• 21. Is dominant, acts as a leader.
• 41. Is full of energy.
• 46. Is talkative.
• 56. Shows a lot of enthusiasm.

Extraversion- Items

• 11.   Rarely feels excited or eager.
• 16.   Tends to be quiet.
• 26.   Is less active than other people.
• 31.   Is sometimes shy, introverted.
• 36.   Finds it hard to influence people.
• 51.   Prefers to have others take charge.

Open-mindedness+ Items

• 10.   Is curious about many different things.
• 15.   Is inventive, finds clever ways to do things.
• 20.   Is fascinated by art, music, or literature.
• 35.   Values art and beauty.
• 40.   Is complex, a deep thinker.
• 60.   Is original, comes up with new ideas.

Open-mindedness- Items

• 5.     Has few artistic interests.
• 25.   Avoids intellectual, philosophical discussions
• 30.   Has little creativity.
• 45.   Has difficulty imagining things.
• 50.   Thinks poetry and plays are boring.
• 55.   Has little interest in abstract ideas.

How surprised should we be that items like “I am inspired to do things” correlate with items like “Is full of energy,” or that “I experience inspiration” negatively correlate with “Has few artistic interests”? “Replication” seems like a dignified term for asking the same questions in different ways.

One of the more surprising correlations replicated by Soto (2019) is between “Agreeableness” (Big Five) and “Heart Disease” (measured by a questionnaire about chest pain). The correlation is only .04, compared to the original .15. Even so, since the most important questions on the 1977 questionnaire ask the subject to agree with various statements about chest pain, rather than phrasing them in the negative, a tiny correlation with agreeableness would not be surprising. It is not clear what scientific hypothesis—particularly a causal hypothesis—might be advanced here. But it counts as a replication.

Soto (2019) reports that Negative Emotionality (sometimes called Neuroticism) correlates strongly with DSM-III Depression and Anxiety. It is hardly surprising that a subject answering questions about negative emotions would answer questions related to depression and anxiety similarly, since many of them are functionally the same questions. It is also interesting that a measure of supposedly stable personality traits correlates with a measure of supposedly pathological disease symptoms. Nonetheless, the correlation is treated as an important replication of the predictive power of the Big Five. 

Many of the “consequential life outcomes” appear to be rephrasings, or aspects, of the personality factors they correlate with: “Existential or phenomenological concerns” correlates with Open-mindedness, “Existential well-being” and “Subjective well-being” correlate (negatively) with Negative emotionality, and “Dating variety” correlates with Extraversion. Interestingly, “Occupational performance” correlates only .03 with Conscientiousness (Soto 2019), even though the questions on the two surveys seem to overlap more than that. Survey statements like “I do a thorough job” or “I tend to be lazy” seem like they should predict job performance, and this prediction is at the core of the claims of proponents of the Big Five. But despite the inanity of the purported correlation, it is not in fact always easy to detect. Salgado’s (2002) meta-analyses, for instance, found that Conscientiousness modestly correlates with “deviant behavior” (such as theft), but does not correlate with absenteeism or accidents. 

On the other hand, Big Five Conscientiousness was not found to correlate with mask wearing in a sample of thousands in Spain during the coronavirus epidemic (Barceló & Sheen, 2020). This was not treated by the authors as any kind of falsification of the Big Five, or even evidence against it. The abstract noun “conscientiousness” has a rich meaning, only part of which is captured by the Big Five, and only a tinier part of which is captured by the two-question methodology used here (“does a thorough job” and “tends to be lazy”). But Conscientiousness is often correlated to health behaviors, and is often said to predict them with various strengths, even though the questions in the survey focus on job performance and tidiness. For instance, Tiainen et al. (2013) found that Conscientiousness correlated with higher fruit intake, at least in women, in a food questionnaire study. (Extraversion was associated with a higher meat intake in women, and each personality trait was found to have some effect on some gender and food type.) Conscientiousness has been linked to the intention to use sunscreen by beachgoers in the UK, although not among beachgoers in New Zealand (Kouzes et al., 2017). Given the loose standards at play, it wouldn’t have been too surprising to discover a relationship between Conscientiousness and mask wearing, or Conscientiousness and any other allegedly health-related behavior.

The methodology of the Spain mask study was poor (asking about mask use without asking about likelihood of leaving the house at all, for example), but the quality of almost all studies in social science is poor, and it is not worse than most studies I have encountered in support of the Big Five. From a naive perspective, it is surprising that poor-quality studies with no practical power to falsify anything are still performed. But this is exactly the genius of the thing. What is interesting is the degree to which the Five Factor Model is insulated from falsification. It makes no causal predictions; others may make predictions about correlations, and these may replicate, or not. The only content of the Five Factor Model, in the sense of making claims, seems to be that the five factors are important somehow, and that their importance is specially validated by mathematical methods using survey instrument responses. That they correlate with responses on surveys ostensibly measuring other things is taken as evidence for their importance, but rarely is failure to find such evidence counted against their validity. There is an old saying that a stopped clock is right twice a day; it is not clear how often the Big Five “clock” is “right.”

As a novice to the study of the Big Five, I found that I had many misconceptions. For example, I thought that the Big Five were considered to be universal, in some sense, and not just descriptive of WEIRD college students. But in fact, it is not simply that the Big Five factors fail to fall out of the analysis of large surveys in other languages and cultures. They don’t even fall out of Big Five-specific questions administered in non-WEIRD populations (Gurven et al., 2013). Many of the concepts mentioned in the Big Five survey instruments do not even exist in non-WEIRD languages as such, particularly abstract nouns (Gurven et al., 2013). Of course a concept that does not exist across cultures could not form the basis for a universally important personality trait, as measured by language.

Second, I imagined that the Big Five were based on large and extremely inclusive sets of survey items. I admit that I hadn’t really thought about the abstract noun “personality.” What counts as personality, and what doesn’t? Apparently, the exact nature of the questions in the survey seem to matter a great deal. Saucier & Srivastava (2015) say that the appearance of the Big Five factors “is clearly contingent on one’s variable-selection procedure,” noting that multiple particularly broad question databases have failed to produce the Big Five as the top five traits. The Big Five are not a property that emerges from all sets of questions describing humans, as the strong claim would have it, but rather from a subset of questions whose nature and selection procedure is not always clear. The consequences of this are unknown.

Third, I’d thought that the Big Five were relatively stable across the life course; however, longitudinal studies of several cohorts of adults, born between 1914 and 1960, revealed that most traits changed over the life course in distinct patterns (Roberts et al., 2006), with social dominance (an aspect of extraversion), agreeableness, conscientiousness, and emotional stability increasing in adulthood and/or later life. One interpretation is that the positive traits identified by the Big Five are traits of adulthood, or reflections of having control over one’s environment, or something like that. 

Finally, I’d thought that the traits themselves were orthogonal, in that they were genuinely separate traits that didn’t covary much. I’d thought that this was a major focus of the factoring process, and an aspect of the traits’ specialness and validity. However, the traits covary a great deal. Lukaszewski et al. (2017) found positive mean inter-factor correlations for every country in a large international sample, and an especially high value for Tanzania, the outlier apparently driving their result (that complex societies decrease trait covariance through specialization). Nor is it unusual to find big correlations between the factors; in a large sample of twins, Shane et al. (2010) found correlations of .39 between Extraversion and Openness to Experience, and .30 between Emotional Stability and Agreeableness. Chang et al. (2012) address the problem of the non-orthogonality of the Big Five, and report that even with a methodological correction to eliminate “methods bias,” they could not eliminate correlations between many factors, suggesting some may be “redundant.”

If the Big Five are not universal, stable, or orthogonal, what good are they? They have a perfectly clear use. They replicate: the answers to many other survey instruments can be found to correlate with the Big Five survey responses, in multiple samples of survey-takers. To complain that the Big Five are meaningless is somewhat unscientific. They have a very specific meaning within the language game they belong to, and they are popular and memetically successful tools within that sphere. 

The Big Five are, in a sense, protected from falsification. They make no predictions; there is no underlying causal model. As I understand it, no study could be devised to prove that the Big Five aren’t real, because they make no formal pretense to reality. They are innocent mathematical constructs that fall out of particular survey instruments administered to particular populations.

Critics of the Big Five are almost always proponents of personality factor models with other numbers attached. Few criticize the connection between the survey instruments and the underlying reality. Consider each survey question. What does it ask the respondent for, other than “a muddle of vague memories of inexact observations,” as Galton put it? Note how every single question may be responded to with “compared to what?” Context, and not just what we might narrowly think of as personality, is relevant in 100% of the questions. Certainly the questions measure something; it is not at all clear what that something is, and the nature of that something is rarely investigated. 

Heene (2013) puts the matter rather strongly, in regard to psychological methods that allow the researcher to slip out of the reach of falsification:

[T]he tools of mainstream psychology such as [Structural Equation Modeling] and [Item Response Theory] make exactly these strong assumptions about the quantitative structure of psychological attributes. But avoiding any tests of quantitative measurement but applying methods making the assumption of quantity appears to be nothing more than a self-delusion that one bears something valuable instead of being in fact empty-handed. This all too strong tendency to avoid falsification is probably deeply rooted in the scientifically unhealthy political/economical aspiration of psychology which keeps the machine for paper-producing and grant-funding well-oiled but also leading to a severe publication bias….[T]he possibly best evidence of my claims comes from a logical argument: has anyone ever seen articles using SEM, IRT, or Rasch models in which the author admitted the falsification of his/her hypotheses? On the contrary, it appears that stringent model tests are mostly carefully avoided in favor of insensitive “goodness-of-fit indices.” (Citations omitted.)

Holtz & Monnerjahn (2017), reviewing psychology textbooks, conclude that Karl Popper’s ideas of falsification have had “little to no traceable influence on the epistemology and practice of social psychology.” Given social psychology’s newfound conversion to the religion of replication, a theory devoid of causal claims, and therefore predictions, is the perfect theory: it can never be falsified. 

But the true power of the Five Factor Model is not just its power to replicate, but its power to bless the measurement of any correlation (or lack thereof) between itself and any construct as valid science. For example, Asselmann & Specht (2020) measured the Big Five traits of subjects multiple times in a longitudinal study, some of whom had children during the study period, and some of whom remained childless. They predicted, based on previous studies, that three traits would change with parenthood, or predict parenthood: conscientiousness, agreeableness, and emotional stability. In fact, they found that exactly the other two traits (extraversion and openness) correlated with parenthood: more extraverted, less open people selected into parenting by a tiny amount, and people became less open and less extraverted after becoming parents. But the authors conclude their abstract triumphantly:

Taken together, our findings suggest that the Big Five personality traits differ before and across the transition to parenthood and that these differences especially apply to openness and extraversion. 

The Big Five survey instrument doesn’t seem to measure the same changes from study to study, but this is taken as support for the results of the latest study. The underlying construct is not questioned. Simply measure the Big Five and report how they correlate with literally anything else, or even with themselves at different points in time, and you’ve performed socially valid science, regardless of your hypothesis and your results. Such is the power of the Big Five.

Is this “better than nothing”? For now, the “better than nothing” of the Big Five is its ongoing use in publishing psychology papers. It is certainly better than nothing for psychology researchers who want to publish research using cheap methods that no one will question (except rude bananas on the internet). But from an outside perspective, the perspective of one who might hope to get knowledge about the world from science, an unfalsifiable theory using methods that evade falsification becoming the dominant paradigm is hardly “better than nothing.” 

Those who are skeptical of the Enneagram are usually Type 6, and those who are skeptical of astrology are usually Tauruses. Similarly, those who criticize the Big Five are typically low on extroversion, high on conscientiousness, low on agreeableness, and high on neuroticism (openness to experience can go either way, I suppose). From the perspective of the social psychology of personality, this essay is a glowing partial replication of the Big Five! 

I predict that the Five Factor Model of personality has many years of life in it yet. Perhaps it will endure for decades, producing replicable findings of dubious significance, and meanwhile crowding out creative and meaningful research into how people work. Perhaps the most important and surprising accomplishment of the Five Factor Model is hiding the fact that such research is not taking place within the field of social psychology.

References

Allport, G. W., & Odbert, H. S. (1936). Trait-names: A psycho-lexical study. Psychological monographs, 47(1), i.

Asselmann, E., & Specht, J. (2020). Testing the Social Investment Principle Around Childbirth: Little Evidence for Personality Maturation Before and After Becoming a Parent. European Journal of Personality.

Barceló, J., & Sheen, G. (2020). Voluntary adoption of social welfare-enhancing behavior: Mask-wearing in Spain during the COVID-19 outbreak. Preprint at OSF https://osf.io/preprints/socarxiv/6m85q/.

Chang, L., Connelly, B. S., & Geeza, A. A. (2012). Separating method factors and higher order traits of the Big Five: A meta-analytic multitrait–multimethod approach. Journal of personality and social psychology, 102(2), 408.

Galton, F. (1884). Measurement of character. Fortnightly, 36(212), 179-185.

Gurven, M., Von Rueden, C., Massenkoff, M., Kaplan, H., & Lero Vie, M. (2013). How universal is the Big Five? Testing the five-factor model of personality variation among forager–farmers in the Bolivian Amazon. Journal of personality and social psychology, 104(2), 354.

Heene, M. (2013). Additive conjoint measurement and the resistance toward falsifiability in psychology. Frontiers in psychology, 4, 246.

Holtz, P., & Monnerjahn, P. (2017). Falsificationism is not just ‘potential’ falsifiability, but requires ‘actual’ falsification: Social psychology, critical rationalism, and progress in science. Journal for the Theory of Social Behaviour, 47(3), 348-362.

Kouzes, E., Thompson, C., Herington, C., & Helzer, L. (2017). Peer Reviewed: Sun Smart Schools Nevada: Increasing Knowledge Among School Children About Ultraviolet Radiation. Preventing Chronic Disease, 14.

Lukaszewski, A. W., Gurven, M., von Rueden, C. R., & Schmitt, D. P. (2017). What explains personality covariation? A test of the socioecological complexity hypothesis. Social Psychological and Personality Science, 8(8), 943-952.

Roberts, B. W., Walton, K. E., & Viechtbauer, W. (2006). Patterns of mean-level change in personality traits across the life course: a meta-analysis of longitudinal studies. Psychological bulletin, 132(1), 1.

Saucier, G., & Srivastava, S. (2015). What makes a good structural model of personality? Evaluating the big five and alternatives. In APA handbook of personality and social psychology, Volume 4: Personality processes and individual differences. (pp. 283-305). American Psychological Association.

Salgado, J. F. (2002). The Big Five personality dimensions and counterproductive behaviors. International journal of selection and assessment, 10(1‐2), 117-125.

Shane, S., Nicolaou, N., Cherkas, L., & Spector, T. D. (2010). Genetics, the Big Five, and the tendency to be self-employed. Journal of Applied Psychology, 95(6), 1154.

Soto, C. J. (2019). How replicable are links between personality traits and consequential life outcomes? The Life Outcomes of Personality Replication Project. Psychological Science, 30(5), 711-727.

Tiainen, A. M. K., Männistö, S., Lahti, M., Blomstedt, P. A., Lahti, J., Perälä, M. M., … & Eriksson, J. G. (2013). Personality and dietary intake–findings in the Helsinki Birth Cohort Study. PloS one, 8(7), e68284.

It’s difficult to study words, because words are hard to see. Words are tools used in communication, and when communication is working, they disappear into invisibility. 

One way to see words is to make a word jail: a list of problematic words ripped out of their contexts, so that they may be seen for themselves instead of hiding behind meanings. 

Another way to see words freshly, to experience them as broken and therefore present, is to enter a new domain with its own unfamiliar jargon. Military basic training, rock climbing, sailing (whether Melville-era or contemporary), and weaving all require that novices take on a new jargon in order to get a grip on a new domain. The jargon enables the initiates to pick out important aspects of the world (in their bodies, in the natural environment, in the technology). With new words, they learn to identify newly-salient aspects of reality and communicate with others about them. 

This is also true in domains with less connection to brick-and-mortar reality. Philosophies and theologies and theories of psychology and politics also use jargons that pick out aspects of reality for consideration, jargons which adherents absorb as they become competent. It seems that there is a difference between the words developed in these domains, and words that develop in the domains in the previous paragraph. The difference is that if words are used wrongly, or their meaning allowed to slip and degrade, in the cases in the previous paragraph, there would be terrible trouble. In the more abstract cases of the present paragraph, it is possible for words to become unmoored without anyone noticing. Perhaps the words even serve to protect these domains from the intrusion of reality. The as-if shared language game becomes the territory itself. 

Such meanings as words get in shared fantasy games are different from the meaning words get in language communities whose classifications and jargon must be relied upon for survival by their practitioners, as in sailing or rock climbing. But even weaving jargon is less degenerate than that of psychology. Apparently even this little amount of contact with reality can protect a jargon from loss of meaning. Weaving language is hotly debated; there is no official terminology in weaving, to the dismay of some deluded would-be standardizers. Perhaps this is an advantage. 

There is, of course, nothing wrong with shared fantasy games. It is only when these games attempt to remake the world, to systematize it according to their own jargons, that problems emerge.

The DSM in psychology is an official book of the presently-dominant school of psychology’s jargon. The words in the DSM have little ability, on their own, to pick out important aspects of reality. The DSM jargon is given meaning when learned in a psychological community (a hospital or practice or university, say) while observing people, classifying them one way and another, until the initiates can aesthetically “see” the different constructs for themselves, just like their teachers. Each observing-teaching community unavoidably develops its own distinct meanings for the DSM words. But within psychology, as within many domains, there is a clear belief that these special jargon words have global meaning: that they apply to humans generally, and that their meanings are clear and shared. This faith, evidenced most in psychology’s tools and experimental methods, is both touching and troubling.

The DSM words grow in population steadily, as if they are constantly found to be inadequate in providing a global picture of the interior human world. It would be impossible for them not to be. 

There is nothing wrong with a word in itself, any more than a word in itself carries meaning. But there are patterns that appear pathological in the way groups use words to pick out patterns, particularly when the words are not the jargon of a community in precarious, direct contact with the world being described. 

To lose faith in words, to find the words broken in general rather than here and there, is an unpleasant deconversion experience. Those who have undergone this crisis in faith—Heidegger, Korzybski, Wittgenstein, Garfinkel—often attempt to leave a record of it, swallowing the irony that they can only communicate using the now-distrusted words. Wittgenstein’s is the most cogent and least brain-melting; the others, and many like them, adopt new jargons, put all kinds of unfamiliar weight on parts of speech, multiply clauses, troll typographically with brackets and scare quotes, and wrap it all in puns, jokes, and absurdities. It seems to be difficult to go through this transition and remain sane, or retain the ability to articulate things in the usual way, if there is a difference between the two.

This traumatic deconversion can only occur among those who have faith in words, and the most pious are the most vulnerable. Even in deconversion, many nonetheless attempt to express the deconversion and its phenomenology in words, new ones used in new ways. It is impossible to know how common the deconversion is. There may be thousands of these idiosyncratic attempts at expression, written from semantic hell, buried in diaries and letters and internet forums. A few deconversion memoirs have been taken up by communities, who learn their jargons, adapt and interpret them as they like, and ensure they live on to confound new generations.

Christopher Alexander’s work exhibits signs of semantic deconversion. The patterns in A Pattern Language, developed in earlier works, especially A Timeless Way of Building, offer names for picking out aspects of well-functioning beauty in the world. They operate similarly to words. Alexander offers them as hopeful replacements for the available building “patterns,” which are ugly and inane. 

Consider the word “fireplace.” It’s not even an abstract noun; it refers to a concrete, functional object. But its meaning can slide and degrade, as the word slips out of entanglement with concrete reality (Timeless Way of Building, p. 235-236):

So long as I build for myself, the patterns I use will be simple, and human, and full of feeling, because I understand my situation. But as soon as a few people begin to build for “the many,” their patterns about what is needed become abstract; not matter how well meaning they are, their ideas gradually get out of touch with reality, because they are not faced daily with the living examples of what the patterns say.

If I build a fireplace for myself, it is natural for me to make a place to put the wood, a corner to sit in, a mantel wide enough to put things on, an opening which lets the fire draw.

But, if I design fireplaces for other people—not for myself—then I never have to build a fire in the fireplaces I design. Gradually my ideas become more and more influenced by style, and shape, and crazy notions—my feeling for the simple business of making a fire leaves the fireplace altogether. 

So, it is inevitable that as the work of building passes into the hands of specialists, the patterns which they use become more and more banal, more willful, and less anchored in reality.

The problem is not with the word “fireplace.” Substituting a new word (“personal fireplace” or “hearth,” maybe) would not solve the problem. The word has shifted meaning because of changes in how material culture is produced. The jargon of psychology will not come to reflect human reality just by adopting new words, either.

In later work (The Nature of Order, Volume 1), Alexander develops a new vocabulary for beauty, an analytic list of fifteen fundamental properties. Alexander strikes a compromise between old and new ways of thinking. He provides a numbered list of named essences like a nineteenth-century scientist, but makes the constructs themselves problematic, operating on each other at all levels. It is an aesthetic task (and a rewarding one) to apply Alexander’s list and absorb the new jargon. Trying to understand them while evaluating beautiful (and ugly) things is worthwhile; trying to understand them while attempting to make beautiful things provides an additional level of understanding. 

Imagine a pair of hikers on the trail, trying to orient themselves at a crossroads. They are looking at a topographic map. One hiker points to a nearby ridge, and then to a feature on the map, and then looks in the eyes of the other hiker. Tacitly, he is saying, “is this ridge here the same as this ridge on the map?”

This vignette serves to illustrate the complexity of the use of words. Sometimes words pick out aspects of reality, like the rock climbing jargon. Sometimes words are names of aspects of a map, model, or theory, like the names of the factors in the Five Factor Model of personality. Sometimes words are even used to analyze the degree of correspondence between the two, as in the unspoken question between the hikers. 

The topographic map is a useful abstraction for the hikers. They are familiar with the method of representing elevation, and a picture of comparative elevations is adequate for them to orient themselves in the terrain. This is to be distinguished from pathological abstraction, abstraction of no use for orientation within an ongoing project. There is a genuine demand for abstraction, for good maps and models, but these are difficult to produce, and degenerate abstraction is easy. Just make a survey instrument and “validate” it, for instance. Then factor the responses, and you have an official, but likely completely useless and meaningless, abstraction of the original idea.

Changing language on its own is of little value. The words are innocent. Losing meaning is merely a symptom of the words coming unmoored from human reality, from making and maintaining, chatting and negotiating. In the domain of architecture, the results have been tragic. Psychology, which has never been particularly good since its inception, has undergone many changes in jargon; the present incarnation distinguishes itself mainly by the degree to which its categories and protocols invade the lives of most people. In politics, abstractions have probably always been dangerous. The famines and horrors of communist revolutions, for example, usually co-occur with lexical and semantic genocides, in which words lose their meaning, and new words, anti-patterns, are introduced (e.g. “kulak”). Again, the words are not to blame. It is the manner by which their meaning is mutilated that is the problem. 

Could there be a widespread crisis of faith in words—a mass deconversion? If it were possible for everyone in the world to simultaneously understand late Wittgenstein, would it be a terrible thing to behold? 

Fact checking of scientific claims by lay volunteers, also known as recreational hostile fact checking or community-based science policing, is a growing hobby. A method for the evaluation of scientific claims by scientifically literate non-expert investigators is presented. The Extended Sniff Test, performed after an initial sniff test, uses the methods of Double Contextualization, Noun Abuse Assessment, and lay literature review, in addition to traditional literature review. As a case study, a suspicious paper is subjected to the Extended Sniff Test, and fails. A quick guide to the Extended Sniff Test is provided.

The paper is available here:

The Extended Sniff Test: A Method for Recreational Hostile Fact Checking

The paper’s “quick guide:”