When AI Closes the Science Gap

What Reproducibility Copilots Reveal About How Research Actually Fails

Time to Complete: 30 minutes 

Download the 5-Minute Warm-Up PDF above 

Who This Is For: This lesson is for anyone who produces, evaluates or depends on published research. That includes academic researchers in biology, earth sciences, computer science and social sciences who suspect that their own published work may not be fully reproducible but lack the tools to find out. It is also for research librarians, data curators and journal editors in open science roles who need a practical vocabulary for auditing the reproducibility of submissions at scale. Graduate students and postdoctoral researchers preparing their first publications face precisely the knowledge gap this lesson addresses, as do science policy professionals and research funders who must evaluate reproducibility claims without doing the computational work themselves. AI product developers building tools for the academic publishing and research verification industries will find the conceptual grounding to design those tools responsibly. The shared problem across all of these roles is straightforward. Researchers externalize only a fraction of what they know, and the gap between what a study contains and what a reader needs to verify it is rarely visible until someone tries to reproduce the work. 

Real-World Applications

Open science mandates from major research funders now require publicly accessible data and code as conditions of publication, yet compliance with those mandates does not guarantee reproducibility. Pharmaceutical and clinical research organizations have learned this directly, spending millions verifying pre-clinical findings that appear reproducible on paper but fail when run in a different laboratory with different preprocessing assumptions. AI-driven reproducibility tools like the one examined in this lesson are already entering journal submission workflows, where they flag missing hyperparameters and inaccessible dataset references before peer review begins. Understanding how those systems work, what they detect and where they fall short is a practical skill for any professional whose credibility depends on the verifiability of published evidence.

Lesson Goal

This lesson builds AI literacy by guiding participants through the design logic of an AI-driven Reproducibility Copilot that analyzes manuscripts, code and supplementary materials to generate structured Jupyter Notebooks and targeted author recommendations. Learners will examine what kinds of reproducibility barriers AI can detect systematically and where human scientific judgment remains necessary. The lesson draws on this study, which demonstrated a reduction in reproduction time from 33 hours to approximately 1 hour. The goal is not to produce AI engineers but informed practitioners who can evaluate what an AI reproducibility tool actually does and what it leaves unresolved. 

The Problem and Its Relevance

Science's reproducibility problem is not primarily a data storage problem, it is a communication design problem. Researchers document their methods for audiences who already share their expertise, yet reproducibility requires documentation precise enough for a reader with no privileged access to the author's tacit knowledge. When that gap is invisible to the author, no volume of open-access mandates or public repositories will close it.

AI copilots expose a structural irony at the center of scientific publishing. A system without contextual understanding treats missing hyperparameter values and undocumented preprocessing steps with equal mechanical attention, while a human expert unconsciously fills those gaps using domain familiarity and never flags them as problems. The same expertise that makes a researcher credible may make that researcher the worst possible judge of whether their own work is reproducible. 

Why This Matters

(i)  Reproduction time is the hidden tax on scientific trust. A study that requires 33 hours to reproduce is a study most researchers will simply accept on faith rather than verify. AI copilots address this directly by reducing that time to approximately one hour, making independent verification a realistic option rather than a theoretical one.

(ii)   The most common reproducibility barriers are not missing datasets. They are missing hyperparameter values, undocumented preprocessing steps and incomplete code that assumes knowledge the reader does not have. These are documentation failures, not data failures.

(iii)  Jupyter Notebooks serve a fundamentally different purpose when structured for reproducibility rather than pedagogy. A pedagogical notebook teaches a method. A reproducibility notebook scaffolds the exact sequence of operations needed to regenerate a specific published result. Conflating the two produces documents that are instructive but not verifiable.

(iv)  AI reproducibility systems benefit both authors and readers through a dual-role design. Authors receive targeted annotations in their manuscript PDFs and code files. Readers receive structured notebooks with placeholders that signal exactly where information is incomplete. The same output serves two purposes at once.

(v)  Coverage matters more than completeness. An AI system that captures all the key figures and tables from a study and flags additional content missing from even pedagogical versions of the same work reduces the practical burden of reproduction substantially, even if some edge cases remain unaddressed.

(vi)  Rote reproducibility and scientific reproducibility are separate problems. Regenerating a figure from its original code and data is a technical task that can be substantially automated. Reconstructing the reasoning behind that figure requires access to the logic and assumptions the authors never wrote down.

(vii)  Open science infrastructure alone does not solve reproducibility. Curated data repositories, community standards and electronic notebooks provide the tools. Ensuring that tools are actually used to document what a reader needs requires active verification, not passive availability. 

Key Concepts

Rote Reproducibility

Rote reproducibility refers to the ability to regenerate the exact figures and tables of a published study using the same code and data. It is the narrowest and most technically tractable form of reproducibility. This is the appropriate initial target for an AI copilot because it is definable and measurable, even though it represents only a fraction of full scientific reproducibility.

Scientific Reproducibility

Scientific reproducibility requires reconstructing not just the outputs of a study but the reasoning, assumptions and interpretive frameworks behind them. It depends on information that authors rarely make explicit and that cannot be recovered from code alone. Research identifies this as the harder long-term challenge, noting that even understanding a paper often requires as much effort as reproducing its computational results.

Jupyter Notebook as a Reproducibility Scaffold

A Jupyter Notebook designed for reproducibility mirrors the logical structure of the original study, provides executable code cells for each major result and uses placeholders to signal where information is missing. It is a guide for the reader rather than a replacement for the author's work. This notebook is generated automatically from the manuscript and updated iteratively as authors respond to copilot recommendations.

The Four Checker Modules

The Hyperparameter Checker identifies critical values that are missing or ambiguous. The Dataset Checker verifies that direct links to all required data are present. The Code Checker detects missing snippets needed for experiment replication. The Documentation Checker evaluates whether inline comments and code structure are comprehensible to a reader outside the author's research group.

Non-Deterministic AI Outputs and Consolidation

Because large language models produce different outputs each time they run, the Bibal et al. system runs each checker five times and uses a consolidation prompt to take the union of all flagged issues. This approach reduces the risk that any single run misses a reproducibility gap. It also means the system's coverage is probabilistic rather than guaranteed, a limitation the authors acknowledge directly.

Dual-Role Design

A dual-role design serves both authors and readers from a single analysis pass. Authors receive annotated PDF highlights and code comments directing them to specific gaps. Readers receive the structured Jupyter Notebook. The system anticipates reader needs by analyzing the manuscript from a reader's perspective, which allows it to generate actionable author recommendations in the same operation.

Coverage vs. Completeness

Coverage refers to the proportion of reproducible content in a paper that the AI system successfully identifies and scaffolds. Research measures coverage by comparing the system's generated notebook against one created manually by a domain expert. Their results show the AI system covers all content included in the expert-created version and identifies additional reproducible elements the expert version omitted. 

Three Critical Questions

Engage with these questions before beginning the activity. Brief written notes will improve your engagement with the steps that follow.

• Can you identify the difference between a paper that is publicly available and a paper that is computationally reproducible, and name at least one specific reason a paper could be the first but not the second?

• If an AI system reduces reproduction time from 33 hours to 1 hour but operates without understanding the science, what does that efficiency tell you about the nature of the reproducibility problem that the 33 hours represented?

• Can you describe a situation in which an AI-generated Jupyter Notebook would make a study appear more reproducible than it actually is, and what kind of human review would be needed to catch that gap? 

Roadmap

The following steps guide you through a structured examination of reproducibility barriers using a specific research scenario. Read all steps before beginning. You have 30 minutes total.

Step 1 — Select a Research Scenario (4 min)

Choose a published study you have read, worked with or are familiar with from your field. It does not need to be a paper you have tried to reproduce. Identify two figures or tables from that study that present computational results rather than diagrams or theoretical illustrations. These two outputs will serve as the targets for the rest of the activity. If you do not have a paper in mind, use any publicly available research article that includes code in a repository.

Step 2 — Audit for Missing Information (6 min)

For each of your two target figures or tables, work through the four categories the Bibal et al. system checks. Identify whether the paper provides the hyperparameter values needed to reproduce that result, direct links to the exact dataset used, all code snippets needed to regenerate the output and sufficient documentation to follow the analysis without asking the author. Record what is present and what is missing. Do not estimate or fill in gaps from your domain knowledge. If the information is not in the paper, mark it as missing.

Step 3 — Sketch the Notebook (6 min)

Outline the structure of a Jupyter Notebook that would guide a reader through reproducing your two target outputs. Identify the sections the notebook would need, where it would insert placeholders for missing information and what explanatory text it would provide for someone unfamiliar with the methods. Note which placeholders you would expect an AI system to generate automatically and which would require author input before the notebook could be used.

Step 4 — Write Two Author Recommendations (5 min)

Based on your audit, write two specific recommendations directed at the original authors of the study you selected. Each recommendation should identify a single gap, explain why that gap blocks reproduction and state concretely what the author would need to add or clarify. Avoid general advice such as 'provide more detail.' Name the specific hyperparameter, dataset reference, code function or documentation element that is missing.

Step 5 — Identify What the AI Cannot Detect (5 min)

Return to your two target outputs and identify at least one reproducibility barrier that an AI checker running on the manuscript and code alone would likely miss. Consider implicit assumptions about data preprocessing, domain-specific conventions that are standard enough that no author would document them and reasoning behind methodological choices that appears only in the author's memory rather than in any text the system could analyze. Name the barrier precisely and explain why it falls outside the system's detection range.

Step 6 — Compare and Discuss (4 min)

Share your Step 5 barrier with another participant or write a response to your own analysis from a different disciplinary perspective. Where your chosen barrier reflects domain-specific tacit knowledge, consider whether that knowledge is in principle documentable or whether it represents a form of scientific understanding that no amount of additional documentation would fully convey. This distinction separates what AI copilots can solve from what they cannot. 

Individual Reflection

After completing the activity, give yourself three minutes to consider the following questions. You do not need to answer all of them.

• How did auditing a specific paper for the four checker categories change your understanding of what reproducibility actually requires beyond making data and code publicly available?

• Did writing targeted author recommendations feel more or less straightforward than you expected, and what does that experience suggest about why authors fail to document these gaps in the first place?

• What would a reader unfamiliar with your field need to understand your paper that you currently do not provide, and how would you begin to close that gap before submission?

• What does the finding that a college student with AI experience could reproduce a study in one hour after copilot intervention suggest about whether reproducibility is fundamentally a technical problem or a design problem?

• If AI systems can automate rote reproducibility substantially, what is the remaining argument for treating scientific reproducibility as a human responsibility rather than a computational one? 

The Bottom Line

Rote reproducibility and scientific reproducibility look like two ends of the same spectrum but they are governed by entirely different logics. Rote reproducibility is a documentation problem that AI can address with mechanical precision because it does not require understanding why a result matters, only whether the information needed to regenerate it is present. Scientific reproducibility requires preserving the reasoning behind a result, and that reasoning lives in the author's judgment rather than in any file a system can read. Treating these two problems as interchangeable is not a technical error. It is a category error, and it produces AI tools that appear to solve reproducibility while leaving its most consequential dimension untouched.

The reproducibility crisis is a symptom of a publishing system that rewards novelty and accepts transparency as an afterthought. AI copilots that flag missing hyperparameters and scaffold Jupyter Notebooks improve the transparency of individual papers without changing the incentives that make that transparency optional. The most important question this lesson raises is not whether AI can reduce reproduction time from 33 hours to 1 hour. It demonstrably can. The more important question is whether speed of reproduction is the right measure of scientific openness, or whether we are again at risk of automating what is measurable while leaving what actually matters to chance.

#AIReproducibility #OpenScienceAI #ScientificVerification #ResearchIntegrity #JupyterNotebooks