Improving trials


Meta analysis is a pass-time for people who don’t do any real work, OK, I’m only joking. However, in ameta analysis, you trawl the literature, analyse the papers and make some suggestions.

Here, it is the trialist, like person A and I guess that includes ProfG too, who don’t know their a**e from their elbow. Apparently they have to ask themselves key questions on every thing. Are the trials big enough? Are the outcomes good enough? and Are the trials long enough?….Why do we mess trials up.

Is it OK to do nothing to wait for the perfact wave? This study looks at MS and Alzheimers and the trialists take a beating.

Now, I am not here to defend the clinicians. They can fight their own battles, but they do what they can.

However, you don’t need a meta analysis and hindsight to know that things are wrong. Need an example? Don’t go to MS, just look at COVID-19.

We have had over half a million dead and more trials than you can shake a stick at and what have we got? …..One single anti-viral, which we can’t get our hands on because a certain country has bought the World’s-Stocks. This has to be administered in hosptial and doesn’t work too well the way it is used. Next we have steroids to stop the issues of severe COVID, but use it too early and it may do the opposite.

I don’t need to do a meta anlalysis… believe that starting an anti-viral, probably 2 weeks, after the infecion has arrived, when the immune system has got rid of the majority of the virus, at a time when the immune system may not be the most important aspect….. to think that it is probably not going to work that well.

It is not rocket science. Doing a bit of reading and common sense may tell us to try anti-virals as soon after infection as possible. However, the funders (UK government) have told us to stay away from Hospital until the “Sh1 has hit the fan” if you get COVID-19

But what do we do in Alzheimers…we do trials after the brain resevere has long been exhausted, because the disease does not show itself until this has happened. Are we are surprised the treatment doesn’t work. You do the easy type of trial first and then hopefully you learn from your mistakes.

Sound familiar? I’m not going to bad mouth the MS trials and trialists, you can do that job.

But is we look at the issues above, I wonder if the authors have had the experience. I haven’t, but I have been looking from the outside and can see some of the problems. There is a suggestion that trials need to be bigger and longer, but it is not that easy.

We have to look at some of the constraints and one of them is the patent system. It works fine for cancer were things happen over a few months and death is a rather hard outcome, that is easy to measure. This is how they discovered steroids in COVID. Thankfully if you have to wait for death in a neurological condition you have to wait a long time. So doing trials in neurological conditions are much more problematic.

The patent system give the company market exclusivity for about twenty years…that is until the lawyers stretch it out. So if you have to develop a drug (Let’s say it takes 7 years), a 5 year phase 2 study (This would take 7 years to do as you have to recruit all those peopl), two five year phase 3 trials (7-8 years), go to the regulators (1 year) to get approval, you have no chance to make your costs back.

Now we do long trials but your are on placebo for 7 years and you feel its not working. Are you happy to keep the trial going that goes on for ever?Now we have the costs. Money does not grow on trees and trials have a cost. In the UK, the government trials can’t cost more that £2-4 millions and yet pharma have to spend £100,000,000 to do the studies to a standard that the regulators expect. So to do an academic trial you may have to fudge the sample size and power to detect change to get within budget. The outcomes have to fit what the regulators want. Don’t do them and you don’t have a drug. That’s OK the academics will do it…..But will they? Can they?. In an area where there is no option they do what they can but once a licenced product is available it becomes less easy.

Ask the question is ther now any part of MS that is not covered by an MS drug. In the US the answer is no. So if the ethical review panel has any backbone and they say it is not ethical to do a placebo but you have to do a comparison to the “best option available” then you have a problem, because unless you use the health system to do that you have to buy the comparator. So if you were comparing drug X in a 500 people trial arm for MS that is about 50,000,000 (say for alemtuzumab remember to at the VAT (20% of costs) for hospital based infusion) a year so 250,000,000 for your trial comparator drug. Use the non best option and say that is a beta interferon thats 3,000,000 a year so 15,000,000 before you start. Also if you are comparing to the best in class you size of the 500 trial is now going to be 1000-2000 person trial as it is harder to show a difference. Are there going to be enough people available to do a trial of that size. With a national health co-ordinated system it may be possible but in other places not so easy. Is this why ethical commitees accept placebo controls?

The paper says the funders should call the shots…but does this mean you do studies, which the populous wants in cohorts of people that the populous wants to be included in the trials? But, what happens when the science says it is easier to do trials in a different cohort of people than the populos wants?

We don’t do that?

What about Stem cell trials? We have done more trials in response to public clamour, without the knowledge, how they can best be used…for example when you get a stem cell how do you know what it is going to turn in to. It is not always a simple binary answer. If ProfG gave you his insight I am sure you would be even more enlightened. It is not always reading.

Learning from Cochrane systematic reviews: what improvements do these suggest for the design of trials?Pirosca S, Clarke M, Treweek S.Version 2. F1000Res. 2020 Mar 11;9:178. doi: 10.12688/f1000research.22635.2.

Background: Many randomised trials have serious methodological flaws that fatally undermine their results, which makes the research wasteful. This is of concern for many, including those doing systematic reviews that include trials. Cochrane systematic reviews have a section called ‘ Implications for research‘, which allows authors of the review to present their conclusions on how future research might be improved. Looking at these conclusions might highlight priority areas for improvement. Methods: We focused on the Cochrane Schizophrenia Review Group and the Multiple sclerosis and rare diseases of the central nervous system Review Group (the MS Review Group). Reviews with citation dates between 2009 and 2019 were identified and the recommendations of review authors in ‘ Implications for research’ were put into categories. Results: Between 2009 and 2019 we identified 162 reviews for the Schizophrenia Review Group and 43 reviews for the MS Review Group. We created 22 categories of recommendations in total, of which 12 were common to both groups. The five most used categories were the same for both: better choice of outcomes; better choice of intervention/comparator; longer follow-up; larger sample size; use of validated scales. Better choice of outcomes and/or intervention/comparator was recommended in over 50% of reviews. Longer follow-up and larger sample size were recommended in over a third, with use of validated scales being suggested in around a fifth of reviews. There was no obvious pattern of improvement over time for trials included in systematic reviews published by both groups. Conclusions: We suggest that trialists working in these and other areas ask themselves, or are compelled to do so by others (e.g. funders), why they have chosen their outcomes, intervention and comparator, whether follow-up is long enough, if the sample size is big enough and whether the scales they choose to measure their outcomes are appropriate

About the author



  • Trialists need to have a thorough understanding of EBM. It’s easy to destroy any publication by review and at some point ‘good enough’ has to be accepted. The great thing is that Cochrane reviews can be critiqued in the same way that reviews ‘critique’ trials. The CASP review tools are excellent for this, but rarely used. When all else fails, bad Cochrane reviewers employ the use of ‘biological plausibility’ to shoot down research that the reviewers cannot fault, cementing their reviewers’ biases in the ‘evidence base’ and setting back research decades in the process. When I did a Cochrane review many years ago, we were told that we had to update the review 2-3 years later, or it would be removed. Unfortunately, Cochrane has double standards when it comes to updating MS reviews that are nearly 2 decades old. Comparing new treatment against placebo is not necessary as NICE rely on the Incremental Cost Effectiveness Ratio (ICER) when deciding what new treatments to support. This can be calculated from studies comparing new treatments against established treatments.

    • Thanks whats the EBM? Electronic body music or Evidenced based Medicine …Cochrane reviews can be critiqued…I am sure we can have and will get a meta analysis of the meta analysis….

      To get to NICE the FDA , EMA, MHRA have to be appeased firt…they care about safety and do not care one bit about cost. On the other hand NICE only care about drug cost, but do not bring the true and whole costs into their equations.

      Anyway you can see from Mark’s comments I don’t really understand the complexities either. I have spent the last few years discussing trials in fact not even that it was which agents to try, you will see a paper soon. Do we wait for the perfect wave or try do something? These trials are going to suck the money out of the MS research budget

By MouseDoctor



Recent Posts

Recent Comments