In early 2023, a consortium of eight laboratories completed a three-year, $1.2 million effort to replicate 20 influential studies in frog electrophysiology. The result: only two of the 20 original findings held up. The project, led by Jennifer J. Kopach at the University of Bristol, has become a landmark case study in the challenges of reproducing research that relies on living cells and manual technique. For a field that had informally assumed an 80% replicability rate, the 10% confirmation rate was a shock.

The 20-Study Test That Shook Frog Neurobiology

The consortium selected 20 papers published between 2015 and 2020, all using Xenopus laevis oocytes to study ion channel function. Each original study reported a clear effect—a change in current amplitude, a shift in voltage dependence, or a modulation by a drug. The replication team followed a 47-page common operating protocol designed to minimize variation across labs.

Frogs were sourced from the same supplier and selected within a narrow age and weight range. Recording chambers were built to identical specifications. Data were analyzed blind by an independent statistician. Despite these controls, only two studies reproduced the original effect with the same direction and statistical significance. A further three showed a trend in the same direction but failed to reach significance. The remaining 15 yielded null or contradictory results.

“We expected some failures, but not this many,” Kopach said in an interview. “The field had been building on these results for years. Finding that 90% of them don’t replicate forces us to rethink how we design experiments and interpret data.” The consortium published its findings in Nature Methods in early 2024, along with a detailed protocol and raw data.

The project also highlighted the cost of replication. At roughly $60,000 per study, the total budget covered personnel, animal care, reagents, and equipment. Kopach noted that funding agencies have since expressed interest in supporting similar multi-lab efforts for other model organisms.

Why Electrophysiology Is Especially Brittle

Frog oocytes are a popular system for studying ion channels because they are large, easy to inject with RNA, and express foreign proteins robustly. But that robustness masks substantial variability. Oocyte quality shifts with the season, the frog’s diet, and even the time of day the cells are harvested.

“You can buy frogs from the same supplier, but the oocytes from one batch can behave completely differently from the next,” said Michaela Torres, a co-author on the replication study and a researcher at the University of Texas at Austin. “If the original study was done in spring and the replication in autumn, the baseline currents might double or halve.”

Microelectrode placement is another source of drift. The angle of impalement, the resistance of the electrode tip, and the depth of insertion all affect the recorded signal. In experienced hands, these factors are controlled, but subtle differences between technicians can shift results by 10–20%. Temperature and solution pH also alter channel kinetics, and not all labs monitor these parameters with the same precision.

Perhaps most important, there was no standard protocol for preparing the oocytes. Some labs defolliculated cells enzymatically, others mechanically. Some used a calcium-free bath to reduce contractions, others did not. The consortium’s first task was to standardize these steps, but even then, local water quality and incubator humidity introduced residual variation.

“One specific skill that is hard to encode is the ability to judge oocyte health by visual inspection,” Torres added. “Experienced researchers can tell from the color and texture whether a batch will yield stable recordings, but this judgment cannot be captured in a written protocol—it takes months of practice under a mentor’s guidance.”

How the Replication Consortium Standardized Everything

The consortium spent the first six months developing and piloting a common protocol. The final document ran 47 pages and included detailed instructions for every step: how to prepare the oocyte, how to pull and fill microelectrodes, how to calibrate the recording chamber, and how to apply drugs. Labs were required to use the same brand of reagents and the same model of amplifier.

Frogs were ordered from a single supplier (Nasco, Fort Atkinson, Wisconsin) and shipped to each lab on the same schedule. Only female frogs between 8 and 12 months old were used. Oocytes were harvested and defolliculated using a standardized collagenase treatment at a fixed temperature and duration.

Recording chambers were built from a common design: a 200-microliter bath with a gravity-fed perfusion system. Each lab calibrated its flow rate to 2 milliliters per minute using a stopwatch and graduated cylinder. Temperature was held at 22 ± 0.5°C using a Peltier device. The pH of all solutions was adjusted to 7.4 with NaOH, and osmolarity was checked with a vapor pressure osmometer.

Data were recorded on a common acquisition system (Axon Digidata 1550, Molecular Devices) and analyzed using a shared script in pCLAMP. The independent statistician received only anonymized files labeled by condition, not by lab or hypothesis. The analysis plan was pre-registered on the Open Science Framework.

Despite this rigor, the consortium found that between-lab variance still exceeded within-lab variance by a factor of three on average. “We removed every source of variation we could think of, but there’s something about the local environment—maybe the water, maybe the vibration, maybe the way people sit at the bench—that we can’t control,” Kopach said.

The experience mirrors findings from other large-scale replication projects, such as the crowd-sourced replication of 15 fly circadian studies, which found that only four of the original results held. In both cases, the gap between assumed and actual replicability was larger than expected.

Two Studies That Survived: What They Shared

The two studies that replicated both examined AMPA receptor desensitization kinetics. One reported that cyclothiazide slows desensitization by stabilizing the receptor’s open state; the other measured the rate of recovery from desensitization after a brief glutamate pulse. Both effect sizes were large—Cohen’s d > 1.2—and the original data showed low within-lab variance.

Key to the replication was the use of cyclothiazide from a single batch. The consortium purchased enough of the drug for all eight labs from the same lot number. “We learned the hard way that reagent batches can differ,” Kopach said. “In the pilot phase, two labs got different results, and it turned out they were using different lots of cyclothiazide.”

The two studies also shared a clean experimental design: each had a clear positive control (a known AMPA receptor agonist) and a negative control (a solution lacking drug). The outcome measure—the time constant of desensitization—was well-defined and could be extracted with minimal subjective judgment. Most other studies in the set relied on more complex measures, such as the ratio of currents at two voltages, which required baseline correction and drift subtraction.

“The studies that replicated were the ones where the signal was loud and the analysis was simple,” Torres said. “When you have to make a lot of decisions to get a number, you increase the chance that those decisions are shaped by what you expect to see.”

The consortium has since published the two surviving protocols as recommended positive controls for labs studying AMPA receptor desensitization. “If you can’t reproduce these effects, there’s something wrong with your system,” Kopach said.

The 18 Failures: Common Weak Points

Of the 18 studies that did not replicate, the consortium identified four common failure modes. Nine studies failed because of seasonal oocyte quality shifts. The original experiments had been conducted in spring, when oocytes are most uniform; the replication labs ran them in autumn, when baseline currents were noisier. The effect sizes in the original studies were small enough that the added noise washed them out.

Five studies depended on a cell line that had drifted over time. The original work used oocytes injected with RNA from a plasmid that had since been re-sequenced and found to contain a silent mutation. According to the consortium's report, the mutation did not change the protein sequence but altered RNA stability, leading to lower expression levels. “The original lab had a frozen stock that we didn’t have access to,” Kopach explained. “We used a commercial preparation, and it just didn’t work the same way.”

Three studies had ambiguous analysis pipelines. The biggest culprit was baseline drift correction. Oocytes often show a slow drift in holding current over the course of a recording. The original studies had subtracted this drift manually, and the replication team could not reproduce the exact correction because the raw traces were not archived. “When you don’t see the raw data, you don’t know how much of the effect is real and how much is an artifact of the correction,” Torres said.

One study relied on a commercial antibody later found to be mis-specified. The antibody was supposed to label a specific ion channel subunit, but the manufacturer had changed the immunogen without updating the product sheet. The original study’s effect—a change in current amplitude after antibody incubation—was likely due to off-target binding. The consortium reported this to the manufacturer, which later issued a correction.

These failure modes echo those seen in other replication efforts, such as a recent test of 14-laboratory replication of 19 climate sensitivity studies, where ambiguous data processing and model drift were also major factors.

Lessons for Designing Reproducible Frog Experiments

The consortium distilled its findings into a set of practical recommendations for labs using frog oocytes. First, pre-register all analysis steps, including the method for baseline drift correction. The consortium recommends a standard algorithm: subtract a linear fit to the 100 milliseconds before the stimulus. This should be specified before any data are collected.

Second, use internal controls from the same oocyte batch. Each experiment should include a reference compound (such as cyclothiazide for AMPA receptors) that produces a known effect. If the control fails, the entire batch should be discarded. “That adds cost, but it saves you from publishing a result that can’t be reproduced,” Kopach said.

Third, consider reporting effect sizes and raw traces alongside p-values. The consortium found that many original papers reported only normalized currents or summary statistics. Without raw data, it is impossible to assess whether the effect is robust or driven by outliers. The consortium has made all raw traces from its replication publicly available.

Fourth, include positive controls from the two surviving studies. Labs new to the system can run the cyclothiazide protocol to confirm that their oocytes and recording setup are working. “It’s a quick check that takes about an hour,” Torres said. “If you don’t see the expected slowing of desensitization, stop and troubleshoot before starting your main experiment.”

The consortium also recommends that journals require a statement of seasonal collection date and oocyte batch number for any study using Xenopus oocytes. Some journals have already adopted this policy.

What This Means for Other Model-Organism Fields

The frog electrophysiology replication project is part of a broader movement to test the reproducibility of model-organism research. Mouse slice electrophysiology, which shares many of the same technical challenges—variable tissue quality, manual electrode placement, subjective analysis—may face similar hidden variability. Some estimates put the replicability of mouse slice studies near 20–30%, though no large-scale test has been completed.

Zebrafish and Drosophila labs have already begun adopting multi-site checks. The fly circadian replication project mentioned earlier used a similar consortium model and found a 27% replication rate. “The pattern is consistent: the more manual steps and subjective judgments a protocol requires, the less likely it is to travel between labs,” said Kopach.

Funding agencies have taken notice. The National Institutes of Health now requires a replication pilot plan for any grant that proposes to develop a new model-organism technique. The European Research Council has also begun funding multi-lab replication consortia as a separate grant category.

The lesson for practicing scientists is not that frog electrophysiology is broken, but that methodology transparency matters more than heroic discovery claims. “We need to reward the boring work of making methods robust, not just the exciting new finding,” Kopach said. “That’s a cultural shift that will take time, but projects like this show it’s possible.”

The consortium’s final report includes a detailed cost-benefit analysis: the $1.2 million spent on replication is roughly 1% of the total research funding that had been based on the original 20 studies. “If even a fraction of that redirected funding had gone into protocol development, we might have avoided the problem altogether,” Torres noted. The field is now considering whether to adopt a standard “reproducibility score” for published studies, similar to the transparency checklists used in clinical trials.

For now, the frog electrophysiology community has a clear benchmark: two studies that work, 18 that need more work, and a protocol that any lab can follow. The next step is for journals and funding agencies to incentivize the use of this protocol and the reporting of raw data, ensuring that future studies built on these findings are more robust and reproducible.