AI | Hallucinations and Illusions of AI in Science

Artificial intelligence is transforming how scientific research is conducted. From assisting with literature reviews to generating synthetic data, AI tools are being incorporated across the entire research process in hopes of accelerating discovery. However, a new perspective paper published in Nature warns that an overreliance on AI comes with epistemic risks that could undermine the very goals of increased productivity and objectivity that AI promises.

The paper’s authors, Lisa Messeri and M.J. Crockett, draw from expertise in cognitive science, epistemology, anthropology and science and technology studies to analyze how AI interfaces with the social nature of knowledge production. They identify four main “visions” that scientists currently have for incorporating AI: Oracle (to search and summarize literature), Surrogate (to generate data), Quant (to analyze large datasets), and Arbiter (to streamline peer review).

While these visions aim to overcome limitations like researchers’ finite time and attention, Messeri and Crockett argue they may backfire by fostering “scientific monocultures.” Monocultures arise when a single approach to conducting research, like relying primarily on AI tools, dominates diverse alternatives. Just as agricultural monocultures are vulnerable to pests and disease, scientific monocultures risk narrowing the scope of inquiry and introducing errors.

Specifically, the authors warn of “monocultures of knowing” that prioritize questions best addressed using AI, marginalizing other valid approaches. “Monocultures of knowers” also emerge by promoting the idea that AI tools can achieve true objectivity by representing all perspectives, when in reality they embed the perspectives of their designers.

But the deepest risks may come from “illusions of understanding” – metacognitive errors where scientists overestimate what they comprehend due to AI. The “illusion of explanatory depth” occurs when predictive models are mistaken for fully capturing phenomena. The “illusion of exploratory breadth” deludes researchers into thinking all hypotheses are considered, not just those AI can test. And the “illusion of objectivity” blinds us to AI’s inherent standpoint.

To build resilience against such risks, Messeri and Crockett emphasize cultivating diversity both in terms of research methods (“cognitive diversity”) and participants (“demographic diversity”). While AI promises efficiency gains that increase productivity, an overreliance on any single approach threatens to artificially constrain the problems science addresses and viewpoints included. The authors argue we must manage new tools responsibly to ensure accelerated output doesn’t undermine genuine understanding.

Looking ahead, they call for evaluating specific AI visions separately and considering how expertise levels affect trust in results. Interdisciplinary teams could foster needed scrutiny. Private industry’s growing role in developing academic AI also raises transparency concerns. Overall, this provocative analysis prompts asking difficult questions about incentivizing high-quality science versus maximized outputs, and balancing humanity’s indispensable cognitive diversity with technological productivity. In an era of profound change, maintaining such balance may be key to maintaining science’s ability to shed light, not illusions.

​

Reference(s)

1. Messeri, L., Crockett, M.J. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024). https://doi.org/10.1038/s41586-024-07146-0

Click TAGS to see related articles :

AI | RESEARCH | SOCIETY