In 2023, a consortium of ten laboratories began a systematic replication test of 18 published mouse olfaction studies. The project, funded by a reproducibility initiative, aimed to assess how well these findings held up under careful independent scrutiny. By the time data collection ended, only three of the original studies produced results consistent with the replication attempts. The rest showed effect sizes smaller than originally reported, often barely distinguishable from noise. The outcome has prompted discussion about the economics and incentives that shape research in model-organism biology.

When a Ten-Lab Replication Effort Begins

The project started with a simple question: how many published findings in mouse olfaction would replicate when tested by multiple laboratories using shared protocols? The 18 studies were selected from a pool of roughly 60 papers published between 2015 and 2020 that examined odor discrimination, pheromone detection, and neural responses in mice. Each study had originally reported at least one statistically significant result with a moderate-to-large effect size.

A multi-laboratory consortium was formed, including researchers from Europe, North America, and Asia. Each lab agreed to run the same experimental procedures, using identical mouse strains, odor concentrations, and behavioral apparatus where possible. The consortium received funding from a private foundation focused on improving research reproducibility, covering equipment, animal costs, and personnel time for two years.

Expectations before data collection were cautiously optimistic. Many of the original studies had been published in well-regarded journals, and the phenomena they described—such as mice preferring a particular odorant or showing neural activation patterns—seemed robust. Yet consortium members privately acknowledged that replication rates in other fields, such as social psychology and preclinical cancer biology, had been low.

The consortium held regular video calls to coordinate protocols and troubleshoot differences. One early challenge was that labs used slightly different versions of the same behavioral apparatus, leading to a decision to standardize equipment across sites. This added several months and roughly $50,000 per lab to the project.

A concrete example of such a challenge came from Lab 4, which had previously used a two-choice chamber with manual odor delivery, while Lab 7 used an automated eight-arm radial maze. Reconciling these designs required building custom adapters that fit both systems, delaying the start of data collection by three months and adding $12,000 in machining costs per lab. This kind of infrastructure friction is rarely accounted for in grant budgets and often goes unreported in the literature.

Why Only Three Studies Survived

When the data were analyzed, the results were stark. Of the 18 original studies, only three produced statistically significant effects in the same direction as the original reports, with effect sizes within a plausible range. Another four showed effects in the same direction but were not statistically significant—what some researchers call a “partial replication.” The remaining 11 produced null results or effects in the opposite direction.

One factor was that effect sizes in the original studies were often much larger than what the replication teams observed. In several cases, the original reported effect was roughly three to four times larger than the replication estimate. This pattern is consistent with publication bias, where only large, surprising effects get published, while smaller or null results remain in file drawers.

Protocol differences across labs also played a role. Although the consortium standardized many aspects, subtle variations—such as the time of day when tests were conducted, the humidity of the testing room, or the handling experience of the experimenter—could influence mouse behavior. The consortium analyzed these variables and found that they collectively accounted for some of the variation, but not enough to explain the failure to replicate.

Statistical significance thresholds were also adjusted. The original studies typically used a p-value threshold of 0.05 without correction for multiple comparisons. The replication analyses used pre-registered analysis plans and applied corrections for multiple testing, which meant that some marginal results no longer met the significance bar.

To illustrate the impact of these factors, consider the original study on odor preference for 2-phenylethanol (rose-like scent) vs. 3-methylindole (fecal-like scent). The original report, published in 2017, found that mice spent 72% of their time in the rose-scented chamber, with a p-value of 0.003 and a sample size of 10 mice. The replication consortium, using 30 mice per group and a pre-registered analysis, found a preference of 54% (p=0.21), not statistically significant. The original effect size (Cohen's d = 1.2) shrank to 0.2 in the replication. This pattern—large initial effect, tiny replication effect—was typical of the non-replicated studies.

The Economics of Mouse Olfaction Research

Conducting a single mouse olfaction experiment involves substantial costs. A standard behavioral setup—including odor delivery systems, video tracking, and computer controls—costs roughly $50,000 per laboratory. Animal housing adds about $20 per mouse per month, and a typical experiment uses 40–60 mice over several months. For the replication consortium, these costs multiplied across ten sites, totaling over $1 million for the two-year project.

Graduate student labor subsidizes much of this work. Many original studies were conducted by PhD students or postdocs who spent months training mice, running experiments, and analyzing data. The opportunity cost of that time is rarely accounted for in grant budgets. When a study fails to replicate, that invested labor is effectively lost.

Grant pressure in molecular biology favors novel, positive findings. Funding agencies typically award grants to projects that propose new discoveries, not to those that aim to confirm existing ones. As a result, researchers have strong incentives to produce surprising results and weak incentives to conduct replication checks. The replication consortium itself was funded by a private foundation, not by the main government science agencies.

The infrastructure for shared protocols and data is still developing. In mouse olfaction, many labs use custom-built equipment and in-house analysis scripts, making it difficult to compare results across sites. The consortium had to invest time and money to standardize these elements, a step that individual labs rarely take.

A trade-off emerges: standardization improves comparability but can reduce ecological validity. For example, the consortium decided to use only male C57BL/6J mice from a single vendor to reduce genetic variation. However, some original studies had used female mice or mixed strains, and it is possible that the original findings were strain- or sex-specific. The consortium’s decision, while necessary for consistency, may have inadvertently eliminated effects that depend on genetic background or hormonal state. This trade-off is rarely discussed in replication guidelines, but it highlights the tension between internal and external validity.

How Incentives Shaped the Original Findings

The publish-or-perish culture in molecular biology encourages researchers to pursue striking results. Journals favor positive, surprising findings, often rejecting replication studies or null results. This creates an environment where small sample sizes and flexible analysis methods can inflate false-positive rates.

Many of the original studies had sample sizes of 8–12 mice per group. With such small samples, a few outlier animals can drive the statistical result. Power analyses suggest that to detect a moderate effect size with 80% power, researchers would need roughly 25–30 mice per group. The original studies were underpowered by this standard, increasing the likelihood that some reported effects were false positives.

No pre-registration requirement existed for these studies at the time they were conducted. Researchers could analyze data in multiple ways, choose which outcomes to report, and decide when to stop collecting data, all without a public record. These practices, sometimes called “researcher degrees of freedom,” can inflate the apparent strength of findings.

Some researchers argue that the mouse olfaction field is not uniquely problematic; rather, it reflects broader norms in biomedical science. A 2021 survey of over 1,500 scientists found that more than half believed there was a significant replication crisis in their field, yet few had ever attempted a direct replication of another lab’s work.

A counter-argument exists: some original authors contend that replication failures can arise from undisclosed differences in animal housing, diet, or microbiome composition, which are hard to control across labs. For instance, the original study on pheromone-mediated aggression used mice from a specific breeding colony that had a particular gut microbiome profile. The replication consortium sourced mice from a different vendor, and the microbial environment may have differed. While this point is valid, it also underscores the need for better reporting of husbandry conditions and the development of standardized microbiological controls—steps that would improve replicability even if they cannot eliminate all variation.

What the Three Replicated Studies Share

The three studies that replicated successfully had several features in common. First, they used robust experimental controls, such as blinding of experimenters to treatment conditions and randomization of trial order. These practices reduce the risk of unconscious bias influencing results.

Second, they had larger sample sizes than the non-replicated studies—typically 20–30 mice per group rather than 8–12. This gave them greater statistical power and made their estimates more stable.

Third, they had already been subjected to independent replication attempts before the consortium project began. Two of the three had been replicated by separate labs as part of earlier, smaller-scale projects. This prior track record made them more likely to hold up in a larger test.

Fourth, they practiced open-data practices from the start. Raw data, analysis scripts, and detailed protocols were publicly available, allowing the consortium to directly compare methods. This transparency made it easier to identify and resolve protocol differences.

One of the replicated studies, for example, examined the ability of mice to discriminate between enantiomers of carvone (spearmint vs. caraway odor). The original authors had uploaded their raw trial-by-trial data to a public repository, along with a detailed protocol for the two-alternative choice task. The consortium was able to replicate the exact analysis pipeline and confirm that the discrimination threshold reported (10% concentration difference) was reproducible across labs. This case illustrates how open practices can facilitate successful replication and build trust in findings.

Lessons for Future Model-Organism Work

The replication consortium offers several lessons for improving the reliability of model-organism research. Pre-registration of study designs and analysis plans could reduce wasted resources by discouraging flexible analysis and selective reporting. Some journals now require pre-registration for animal studies, but the practice is not yet widespread.

Multi-lab replication designs, like the one used here, could become standard for high-impact findings. Instead of a single lab running a small experiment, a distributed network of labs could test the same hypothesis simultaneously, providing a more robust estimate of the true effect size. Funding agencies would need to support such infrastructure, which currently falls outside typical grant mechanisms.

Funding agencies should also reward replication work. A few initiatives, such as the Reproducibility Project: Cancer Biology, have demonstrated that replication studies can be conducted at scale, but they remain rare. Shifting a small fraction of grant money from novel discovery to replication could improve the overall reliability of the literature.

Infrastructure for shared protocols and materials is needed. The consortium created a repository of standardized behavioral protocols, odorant concentrations, and analysis code, which is now available to other researchers. Such resources reduce the cost of future replication attempts and make it easier for new labs to enter the field.

The Cost of Ignoring Replication

The financial cost of the 15 non-replicated studies is difficult to calculate precisely, but rough estimates suggest that the original research consumed roughly $400,000 in direct funding, plus uncounted graduate student labor. The replication consortium itself spent over $1 million. Together, these figures represent a substantial investment in findings that did not hold up.

Research time lost is even larger. If each original study required one to two person-years of effort, the 15 non-replicated studies represent roughly 15–30 person-years of work that did not contribute to a reliable knowledge base. The replication effort itself consumed about 10 person-years across the ten labs.

Trust in olfaction findings erodes slowly but persistently. Researchers who built hypotheses on the non-replicated results may need to revisit their own work. Clinicians or companies developing odor-based diagnostics or repellents based on these findings could waste further resources. The field’s reputation for rigor may suffer, making it harder to attract funding and talented scientists.

Career incentives remain misaligned with rigor. Young researchers still face pressure to produce positive, novel results for tenure and promotion. Until institutions reward replication attempts and null results, the cycle of low replicability is likely to continue. The consortium’s findings are a reminder that scientific progress depends not only on discovery but also on the hard, often unglamorous work of checking what we think we know.

Beyond the immediate costs, there is a broader opportunity cost: the research community could have invested the same money and effort in studies that are more likely to yield robust, actionable knowledge. For example, the $1.4 million spent on the consortium and original studies might have funded a large-scale, multi-lab investigation of a single, well-validated phenomenon—such as the neural basis of innate odor aversion—that could have produced definitive results. Instead, the field now has a mixed bag of findings, many of which are likely false positives. This is not an argument against exploratory research, but rather a call for a more balanced portfolio that includes rigorous replication as a routine component.