In the quiet corners of the internet, a new kind of phantom has emerged: a disease that does not exist. Over the past several months, a purely fictional medical condition began appearing in the responses of conversational AI chatbots. What started as a digital hallucination — a typical quirk of large language models, which generate plausible-sounding text without any grounding in factual accuracy — soon gained an alarming degree of legitimacy, eventually making its way into the pages of a peer-reviewed medical journal.
The incident, reported by Le Monde Sciences, exposes a vulnerability that sits at the intersection of two powerful trends: the proliferation of AI-generated content across the open web, and the increasing reliance of researchers, editors, and indexing systems on digital sources that may themselves be contaminated by that same content. The result is a recursive loop in which fabricated information is laundered through layers of apparent credibility until it becomes nearly indistinguishable from established fact.
The anatomy of a feedback loop
Large language models do not retrieve information the way a search engine queries a database. They predict the next most likely token in a sequence, drawing on statistical patterns absorbed during training. This architecture makes them fluent but unreliable narrators: they can produce text that reads like a medical textbook entry while describing a condition that has never been observed in a single patient. The phenomenon is well documented in AI research and is commonly referred to as "hallucination."
The problem compounds when hallucinated content escapes the chatbot interface and enters the broader information ecosystem. Blog posts, forum answers, preprint repositories, and even lightly moderated reference sites can pick up AI-generated claims and reproduce them without verification. Once indexed, these secondary sources become training data or reference material for subsequent models and human researchers alike. Each cycle of absorption and reproduction adds a thin layer of apparent legitimacy. By the time a fabricated disease name surfaces in a manuscript submitted to a journal, it may arrive accompanied by what looks like a trail of corroborating references — references that, upon close inspection, trace back to the same algorithmic origin.
This dynamic is not entirely without precedent. Wikipedia's experience with "citogenesis" — the process by which an unsourced claim on the platform gets cited by an external publication, which is then used to source the original Wikipedia entry — offered an early warning of how circular referencing can manufacture false authority. The difference now is one of scale and speed. Automated systems can generate and propagate content orders of magnitude faster than any human editor can audit it.
The scientific record under pressure
The infiltration of a fictional pathology into a professional medical journal raises questions that extend well beyond a single retraction. Peer review, the principal gatekeeping mechanism of academic publishing, was designed to evaluate methodology, originality, and logical coherence. It was not built to detect whether a cited condition exists in clinical reality — a check that, until recently, would have been redundant. The episode suggests that the assumptions underpinning editorial review may need to be updated for an era in which the raw material of scholarship can itself be synthetic.
The challenge echoes famous episodes in the history of scientific fraud, from the Piltdown Man forgery in paleoanthropology to the Sokal affair, in which a physicist submitted a deliberately nonsensical paper to a cultural studies journal to test its editorial standards. Yet those were acts of intentional deception. The current case appears to be a byproduct of system design — no hoaxer needed, only an architecture that optimizes for fluency over truth and an ecosystem that mistakes fluency for authority.
For journal editors, the immediate implication is operational: new verification layers may be required, potentially including automated checks against established medical ontologies and clinical databases. For the broader research community, the implication is more unsettling. If the archives that future models train on are already seeded with hallucinated content, each generation of AI risks compounding the errors of the last. The scientific record and the algorithmic systems that increasingly mediate access to it are now locked in a relationship where the integrity of each depends on the other — and neither is currently equipped to guarantee it.
With reporting from Le Monde Sciences.
Source · Le Monde Sciences
