Two Thousand Years of Peer Review

Long before anyone called it peer review, communities that needed to know which claims to trust kept reinventing the same test. It set the persuasiveness of the argument aside and asked who stood behind the account, what their record was, and what they stood to lose if it proved false. That test has been run in laboratories, courtrooms, and newsrooms. One of the oldest continuously operating versions of it is the church, which has been putting the question to its own teachers for two thousand years.

In the winter of 144 CE, a wealthy shipping merchant from the Black Sea port of Sinope (on the northern coast of modern Türkiye) presented himself to the church in Rome, made a substantial donation (the tradition records twenty thousand sesterces,^[1] though the exact sum is disputed) and announced that the community was teaching its scriptures wrong.^[2] His name was Marcion. His complaint reached far past a few mistranslated passages, all the way to the entire enterprise of reading the Hebrew Bible as Christian scripture, on the ground that the God described in those texts (vengeful, tribal, obsessed with law) was a lesser deity, a different being from the Father whom Jesus revealed. This required, he concluded, a drastically edited canon: one cleaned-up gospel, ten pruned letters of Paul, everything else discarded as belonging to a different religion.^[3]

The Roman community expelled him. What followed, across the next half century, mattered less for what it settled about Marcion’s theology than for the method by which it was settled.

Irenaeus of Lyon, writing around 180, met Marcion with something other than a superior reading of the disputed texts. A textual counter-argument was available and would have failed, because Marcion was a skilled exegete who had already anticipated the obvious moves. What Irenaeus assembled instead was a set of chains.^[4] He named the men who stood at the head of the major Christian communities and traced their teachers backward: Polycarp of Smyrna, whom Irenaeus had known as a child, had sat at the feet of John the apostle; Clement of Rome had been instructed by Peter and Paul; the churches of Ephesus and Corinth could point to living memory of the apostles. The teaching he defended carried its authority, he argued, by descent rather than by interpretation. It had been handed down, person to person, by teachers who could be named, whose instruction could be checked against what those same teachers had taught years earlier in other cities, and whose standing in their communities depended on staying faithful to what they had received.

Marcion’s credential was a reading of an edited text. Irenaeus’s credential was a chain of named witnesses who could be checked against one another and would have been exposed if they taught one thing in Ephesus and another in Rome. The compelling argument was not “our theology is more accurate,” but “our account has people behind it who can answer for it.”^[5]

Set aside whether Irenaeus was right; some of the chains he asserted are insufficiently documented.^[6] The point is the instinct itself: when a fundamental claim about reality is disputed, ask who stands behind the account and what they lose if it is false, before asking which is the more compelling. The order is not a claim about where truth ultimately rests; it is a claim about how a disputed account gets adjudicated among people who do not yet agree.^[7] The Marcion controversy made that the operating principle of a community that would go on to run the same question through every later dispute, in thousands of distinct and often disagreeing branches, for two thousand years. Whatever else it was, the early church was a testimony-verification system. Inefficient and fallible, but one that kept asking, with remarkable consistency, who is behind this account and what do they stake on it. The grammar was instrumental, but the thing it served was not. The church checked the testimony so carefully because of what the testimony was about, a claim it held to be revealed rather than generated, and Marcion was expelled for what he taught about God and the goodness of the Creator, not for a defective chain of custody. The verification served a truth the community believed it had received rather than invented.

The councils that codified its conclusions (Nicaea in 325, Constantinople in 381, Chalcedon in 451) are often taught as power politics in theological dress, and they were partly that.^[8] Structurally they were also something else: events where named people staked their standing, in public, on a position that excluded others and could be held to it. A creed is a costly signal. Arius, condemned at Nicaea, lost his bishopric and was exiled; Athanasius, who defended the council’s conclusion, was exiled five times as emperors sympathetic to the Arian position reversed the finding and the tables turned.^[9] Both sides paid as the political wind shifted; neither was exempt from the mechanism.

The obvious objection, that this is just motivated persistence (communities holding beliefs for a thousand years out of inertia and identity rather than truth), is not the point here, what is the point is that cost verifies the seriousness with which a claim is held, while leaving its content open. Cost screens out the liar and the indifferent; it cannot decide between two sincere parties, which is exactly why a believer’s concern with truth and this chapter’s concern with testimony never collide.^[10] That is the distinction that matters in a world where producing the appearance of belief costs nothing at all.

---

The same architecture, built under the same pressure through an entirely independent route, runs the institution that defines credibility for secular modernity: the scientific literature.

A paper is claims with authors attached, people who stake their professional standing on the methodology and the results. Peer review is the community of qualified judges, the bishops checking the teaching; retraction is the excommunication, the formal withdrawal of the certification that this account met the community’s standards; citation is the isnad, the chain by which a finding is linked back to the named sources and original observations that support it. The vocabulary is secular and the gowns are different. The mechanism is identical.

The reproducibility crisis of the 2010s was, in structural terms, an accountability failure. Statistical practice allowed the publication of findings unlikely to replicate; the incentives to publish diverged from the incentives to be right; and the system had almost no efficient mechanism for catching fabrication. Diederik Stapel, a Dutch social psychologist, fabricated data across dozens of studies and slipped past ordinary peer review, which checks the logic of an argument while taking the reality of the data on faith. When the chain was finally followed (graduate students who noticed inconsistencies, a committee that pulled the original files, the failure of other labs to reproduce findings no one had actually produced) the mechanism that exposed him was exactly Irenaeus’s: named people, making specific claims, whose testimony could be checked against independent replication and against what they had claimed before.^[11] Stapel published fabricated data across fifty-five studies over more than a decade, collected prizes for invented work, and trained students whose careers rested partly on foundations they had no way to know were false. The system corrected, through the same distributed accountability, but only after a decade of damage, during which a field absorbed assumptions shaped by results that never existed. The architecture corrects after the fact rather than preventing in advance, and a great deal slipped through before anyone pulled the thread. The reforms that followed (pre-registration, open data, mandatory replication) are all variations on the same demand: show the chain. Who collected this data? Who can reproduce it? What do they lose if it does not hold?

Science and the church reached the same design from opposite directions under one pressure: claims proliferate faster than any individual can check, and in a world of abundant confident assertion, the only durable signal is cost indexed to a name. And both crises, the ancient and the modern, were resolved the same way, slowly and contentiously, by distributing the accountability across enough independent actors that no single center of power could suppress the reckoning.

---

But both systems carry the same failure.

Accountable institutions have an appalling record of catching their own leaders. The celebrity pastor who falls years too late. The laureate whose fabrications run unchecked through a field for a decade. The executive whose numbers held only because the auditors answered to the same board. The pattern recurs across traditions, centuries, and cultures with enough regularity to count as a structural risk of any testimony-based system, religion included. So the objection is fair: if the system is so accountable, why do so many powerful people go uncorrected for so long?

The answer turns entirely on separating the design from its common failure mode, and it leaves every particular record to stand or fall on its own. The design works when the community judging the testimony is genuinely independent of the source it is judging, has the knowledge to detect error, has real exit options, and is staffed by people whose positions survive the source’s disapproval. Marcion was adjudicated, however slowly, because the communities doing the adjudicating were distributed and not controlled by any single center. Stapel was caught because the field was large and decentralized enough that independent labs could run the experiments and publish different results. The failure mode is the exact inversion: when a community’s identity fuses with a leader’s authority, exit gets costly; when the only people positioned to question the leader depend on his approval, the mechanism does not merely fail, it actively protects him.

This is why the best-designed accountability systems are deliberately adversarial: the prosecutor does not answer to the defendant, the external auditor does not answer to the CFO, the appellate court does not answer to the trial judge. The failures confirm the design from the negative side, clustering precisely where independence breaks down, exit gets costly, and the checking community is captured. The design is right. The implementation has always been uneven. Those are different things.^[12]

---

Set every theological and scientific claim aside, and the structure still stands. For two thousand years a global network of communities has been running a testimony-verification system on one question: whose account is this, who taught them, does what they say comport with what others who received the same tradition say, and what do they lose if they are wrong? It has produced councils and creeds, peer review and replication, schisms and priority disputes, retractions and catastrophic failures. It is neither efficient nor unbiased. But it has run continuously, across thousands of separate communities, for twenty centuries, with the scientific tradition running in parallel for three hundred years more, and neither has found a substitute for the same basic requirement.

What both have always understood, and what the information economy is now being forced to understand at speed, is that the testimony cannot be separated from the cost. An account whose source pays nothing to make it is a different kind of thing from an account whose source pays something. The payment leaves the content exactly as true or false as it was (the martyrs were sometimes wrong, and cheap content is sometimes right); what it does is signal, in a world of abundant assertion, that something real stands behind the words. You can fake the appearance of cost for a while. You cannot sustainably fake cost itself, because it requires an ongoing exposure that either continues or is revealed to have stopped.

The information economy is rediscovering this under market pressure right now. Value moving to named individuals rather than mastheads (the 404 Medias, the newsletter writers with a million subscribers and no institution behind them) is the economy pricing the same signal the bishops priced when they demanded apostolic chains. The provenance protocols being built into camera firmware and the C2PA standard are isnad systems: digital chains of custody, engineered so that breaking the chain leaves a detectable mark.^[13] The courtroom’s insistence, refined over four centuries, that testimonial statements require a body that can be cross-examined, is Irenaeus’s argument written into a legal doctrine that predates generative models and will not yield to them.

Notes (13)

1. Twenty thousand sesterces represented roughly five thousand days of unskilled labor at Roman wages: the equivalent, by rough modern analogy, of somewhere between $100,000 and $300,000. The figure is disputed; some sources report 200,000 sesterces. ↩︎

2. Marcion of Sinope arrived in Rome around 140–144 CE and was eventually excommunicated in 144. The tradition (preserved in Tertullian and Epiphanius) records that he made a substantial donation to the church, variously reported as 20,000 or 200,000 sesterces, which was returned upon his expulsion. Tertullian, De Praescriptione Haereticorum 30; Epiphanius, Panarion 42; Britannica, “Marcion.” ↩︎

3. Marcion’s canon comprised a single gospel (a version of Luke stripped of its birth narratives and Jewish-friendly elements) and ten Pauline letters, collectively called the Evangelion and the Apostolikon. This is the earliest known attempt to produce a defined Christian canon. Harry Y. Gamble, The New Testament Canon: Its Making and Meaning (Fortress, 1985). ↩︎

4. Irenaeus of Lyon, Against Heresies (Adversus Haereses) III.3.1–4, written approximately 180 CE. New Advent, Ante-Nicene Fathers I. ↩︎

5. The orthodox case against Marcion was never only a chain-of-custody argument, and a reader who takes the authority of the text seriously is right to insist on this. Irenaeus argued from apostolic succession (Against Heresies 3.3) and from the shape of the scriptural canon itself, including his defense of the fourfold gospel (Haer. 3.11.8); and Tertullian’s five books Against Marcion are overwhelmingly a textual argument, that Marcion had mutilated Luke and Paul and severed a revelation the two Testaments hold together. The text indicts Marcion. The narrower point here is that apostolic succession gave that textual argument a checkable social location, a way for third parties who did not yet agree to verify who had altered what. The argument from Scripture and the argument from the witnesses worked together, the second making the first auditable. ↩︎

6. The specific succession claims Irenaeus makes, particularly the Polycarp-to-John connection and the Roman bishop list, are attested primarily by Irenaeus himself, writing decades after the fact, and have been questioned by historians including F. F. Bruce and Bart Ehrman. The historical skepticism concerns the accuracy of specific chains, not the method itself, which is what this chapter is concerned with. ↩︎

7. A reader in the Reformed tradition will object that this inverts the proper order: Scripture is autopistos, self-authenticating, its authority sealed by the internal testimony of the Holy Spirit and resting on God rather than on the testimony of any man or church (Calvin, Institutes 1.7.4-5; Westminster Confession 1.4-1.5; Michael Kruger, Canon Revisited, 2012). The objection is well taken, and nothing here denies it. The claim is instrumental rather than ontological. It describes how disputed claims get adjudicated among parties who do not yet share the conviction, not the ground on which Scripture’s authority rests for those who hold it. The two operate at different levels and do not compete: one answers why a thing is true, the other how a community sorts rival accounts when sincerity alone cannot. ↩︎

8. The Council of Nicaea (325 CE) was convened by Constantine and produced the original Nicene Creed, addressing the relationship of Christ to the Father and condemning the position associated with Arius. Henry Chadwick, The Early Church (Penguin, 1967), remains the standard accessible account. The imperial politics were real: Constantine, sole emperor only since 324, wanted religious unity and presided over the proceedings, and the conciliar process was shaped throughout by shifting alliances and imperial favor. The serious scholarly treatments do not deny this; they argue that the political pressure and the theological reasoning ran together rather than the former simply manufacturing the latter, with the doctrinal questions retaining their own weight independent of who held power at any given moment. Lewis Ayres, Nicaea and its Legacy: An Approach to Fourth-Century Trinitarian Theology (Oxford University Press, 2004), is the standard scholarly account of how the theological and political dimensions intertwined; on the broader entanglement of Christianity with imperial power in this period, see Ramsay MacMullen, Christianizing the Roman Empire (A.D. 100–400) (Yale University Press, 1984). ↩︎

9. Arius was exiled under Constantine after the council’s initial condemnation. Athanasius, who defended the Nicene position, was exiled five times as the political pendulum swung toward emperors sympathetic to the Arian position (Constantius II, Valens). Both the condemned and the defender were exiled at different moments by different emperors; the point being that the cost of holding a position was real for all parties, not only for the losing side. J. N. D. Kelly, Early Christian Doctrines (A&C Black, 5th ed., 1977). ↩︎

10. The distinction has to be kept clean, because it is the sharpest objection here. Cost is evidence of sincerity, the seriousness with which a claim is held, not evidence of its content. People have died for contradictory claims; martyrdom (Greek martys, witness, before it ever meant one who dies) screens out the liar and the indifferent but cannot adjudicate between two sincere parties (Tertullian, Apologeticus 50, on the blood of Christians as seed). This is why the chapter says cost verifies seriousness “while leaving its content open.” A believing community’s concern with truth and this book’s concern with testimony are not rivals: the testimony test runs first, narrowing the field to accounts someone will pay to stand behind, and the question of truth is fought out among those. ↩︎

11. Diederik Stapel was a professor of social psychology at Tilburg University whose data fabrication across at least 55 published studies was exposed in 2011 by junior researchers who noticed statistical anomalies. The subsequent investigation is documented in the Levelt Committee Report (2012). ↩︎

12. The same test falls, eventually, on the makers of the models themselves. A lab that trains and ships a system is a source making claims about what it built and how it behaves, and the question this chapter puts to every authority applies to it unchanged: who is independent enough to check it, what happens to them if they cross it, and is there an exit. The conclusion returns to this directly, arguing that an unaccountable lab and a nationalized frontier are the same danger in different uniforms unless someone outside can drag either into the light and make it answer. ↩︎

13. The Coalition for Content Provenance and Authenticity (C2PA), founded in 2021 by Adobe, BBC, Intel, Microsoft, and others, has developed a specification for cryptographically signing content with provenance metadata. C2PA.org. ↩︎