No publication without reproduction

Mogil JS & Macleod MRNo publication without confirmation

This says
“Concern over the reliability of published biomedical results grows unabated. Frustration with this ‘reproducibility crisis’ is felt by everyone pursuing new disease treatments: from clinicians and would-be drug developers who want solid foundations on which animal studies are based”. 

Yep…you say animal experiments are a load of old mushroom food. 

Can we decipher what this is saying?

“Tightening rigour across all experiments will decrease the number of false positive findings (false claims), but comes with the risk of reducing experimental efficiency and creativity”.

If you put quality control in animal experiments, a lot of the unreproducible guff that you read will be debunked as nonsense

“Bolder ideas are needed”

Listen to me

“What we propose is a compromise between the need to trust conclusions in published papers and the freedom for basic scientists to explore and innovate”

We want you to do good old fashioned science!

“It is proposed that a new type of paper for animal studies of disease therapies or preventions: one that incorporates an independent, statistically rigorous confirmation of a researcher’s central hypothesis”. 

“The confirmatory study would have three features”. 

  • First, it would adhere to the highest levels of rigour in design (such as blinding and randomization), analysis and reporting. 
  • Second, it would be held to a higher threshold of statistical significance, such as using P values of P < 0.01 instead of the currently standard P < 0.05. 
  • Third, it would be performed by an independent laboratory or consortium. 
  • This exceeds the requirements currently proposed by various checklists and funders, but would apply only to the final, crucial confirmatory experiment.
“Enough animals should be studied so that a positive statistical test means that the hypothesis is very likely to be correct”. 

Let’s get rid of the crap that some people are peddling and get some independent support for the wacky ideas. 

If they are shown to be crap we can throw them in the bin.
  • Sample sizes for this crucial experiment would need to go up so more animals used; they estimate around sixfold.
     Overall, however, the subsequent savings in both animals and money are likely to be substantial; fewer people would waste resources following up on weak papers.

The literature is full of weak papers that will never reproduce

“Preclinical trials should be run by researchers with strong expertise in the relevant animal models, and it was believed that some will decide to specialize in performing confirmatory experiments for colleagues”.

Yep independent people may get an answer. But not sure if expert is expert. Nine crap experiments using dogey EAE model that every one agrees on, verses one decent model could be an exception.
This is the problem in EAE research

“We call this large confirmatory study a preclinical trial”. 

However, you took nine EAE labs you may get nine crap experiments and as they would be the majority you would do it there way. This is the current way it is done EAE by numbers with a prescriptive design and analysis.

“These would be more formal and rigorous than the typical preclinical testing conducted in academic labs, and would adopt many practices of a clinical trial”.

Does this mean they will be sent on interesting jollies to exotic locations like clinicians:-) 

However, the authors have made a mistake. Most preclinical studies aim to describe a mechanism and have no thought towards therapy, even when it is dressed up as therapy

“Another option would be to establish dedicated animal-testing facilities, analogous to genomics and bioinformatics core facilities”. 

Em….These are called contract research orgnaisation (CRO), which pharma  already use to independently check  the activity of drugs in animals.

Consortia may be set up

Gravy train for academics and this is the issue here. it is an idea of an existing (stroke) consortia looking for people to buy into the idea. 

The central issue (above as I was a witness to the discussions) that was not grasped in my mind was time in most human studies you dont get treated for hours but in the animal experiments they put the drug in before the stroke, so nine crap stroke approaches does not give you a good one. If you put your drug into animals hours after the stroke they dont work, so nobody wants to publish negative data.

This  not needed as we have CRO doing the job?

“Specialized confirmatory labs would increase the quality of the animal studies, and free the labs that did the initial experiments to focus on their core expertise”.

I suspect you will find that most animal studies do not reproduce

It is thought that “government funders and industry partners, which have spent billions of dollars on disappointing clinical trials, would be prepared to shift resources to support such an improved system, perhaps by offering dedicated grants”.

Please give our wacky ideas cash.

Confirmatory labs would be less dependent on positive results than the original researchers, 

CR doesn’t need to cheat

For findings with immediate clinical applications (that is, a potential treatment that might go into human testing),  such studies are more likely to provide clinically useful information and to survive replication attempts.

Yeah. A multi centre animal clinical trial. Costing millions?

However, as Keith Lemon says “Shittin” ..I say “shitout” 

It is only as good asa the testing facilities

“This system would slow the rate of publications, but not the pace of discovery. Scientific priority could be established by the date on which an experimental plan was agreed (essentially ‘registered’) between the original researchers and those performing the confirmatory study. Furthermore, if published studies are more reliable and public confidence in science is boosted, a somewhat slower publication process seems acceptable. We trust that reviewers and tenure committees will find appropriate ways to credit papers that include confirmation”.

Ho Ho Ho….Papers which are not novel…death wish!

The first question not asked is. Will pharma buy into this? 

Because if they don’t it is a waste of time.

Multicentre trials..Pharma are the only show in town that can afford them, but a good independent lab is worth more that a load of useless experiments 

Pharma already have superior experimental design in terms of randonisation and blinding compared to academics because they work to better laboratory standards than academics. Likewise, as part of their due diligence programme they use independent CRO to do the work and ask that more than one model is used to show efficacy. 

So is this paper asking grandma to suck eggs? 

So the question posted is should we publish?
The problem is there is no quality control across journals and so if one journal rejects there is always another to accept.

However the simplest way to get quality is in the patenting arena. 

Pharma would welcome good quality science to file a patent because they adhere to this standard. However if academica wants to file a patent then they need have to achieve minimal standards and gain quantitative approval 

About the author


Add comment

By MouseDoctor



Recent Posts

Recent Comments