Key concept: Non-inferiority
To understand the concept of non-inferiority in medical research, it's helpful to remember Miller Lite's long-running slogan "great taste… less filing."
Miller Lite was the first successful light beer. And, to persuade people to try it, advertising executives had to find a way to convince people that it would taste as good as regular beer, but that it had added benefits.
The tag line for these spots was "Lite Beer from Miller: Everything you've always wanted in a beer. And less."
Non-inferiority studies are aiming to prove essentially the same thing. They aim to show that a new drug will work about as well as standard treatments but that it comes with other important advantages – it costs less, it has fewer side effects or it's easier to take, for example.
So what exactly does non-inferiority mean? It might sound like it means one drug is "as good as" another. But statistically, when a drug is non-inferior to another treatment, it actually means something a little different: “not much worse than.”
There are a number of reasons researchers might go for non-inferiority over outright superiority. For example, in cases where it would be unethical to compare a treatment to a placebo, leaving one group of sick patients untreated.
But non-inferiority trials also require careful interpretation, and if researchers don't exercise that care, it may be up to the reporter who's covering the study to spot and question the flaws.
Non-inferiority trials can have five different outcomes. Here's a handy graph, from a great paper on non-inferiority by Schumi and Wittes that was published in the open access journal Trials in 2011.Before researchers conduct a non-inferiority study, they choose a range of results, called the non-inferiority margin, which would prove a tested treatment to be “not unacceptably worse” than a standard drug. The non-inferiority margin is represented by the dotted line in the graph. The solid vertical line, at zero, represents no difference between the treatment results of the test drug and the control. Results that fall to the right of zero are better than the standard drug.
Results may fall outside the non-inferiority margin either because the new drug worked better than, or was superior to, the standard treatment, or because the drug didn't work as well, or was inferior. (Note that test drugs can still be inferior to the standard treatment and also pass the test for non-inferiority if their confidence intervals don't cross the non-inferiority margin but also don't meet zero. That's the next-to-last example in the graph. In those cases, newer drugs don't work as well as the standard therapy, even if they also meet the definition of non-inferiority.)
Here's how one non-inferiority test played out in a recent study in the New England Journal of Medicine. The study compared the newer clot preventing drug Apixaban to the standard therapy with two older blood-thinning medications, heparin and warfarin, for the treatment of venous thromboembolism—dangerous blood clots that form in veins.
Study authors weren't sure, going in, that Apixaban would be more effective than standard therapy, which is pretty effective. But it didn't have to be. Because it doesn't require the frequent checkups and constant dosage adjustments that are part and parcel of taking warfarin, it's much easier for patients to take. That's enough to convince many doctors to try the newer drug over the old.
Before the study started, researched defined non-inferiority margins. These are stated in the study's methods section:
The study was designed to test the hypothesis that apixaban would be noninferior to conventional therapy with respect to the primary efficacy outcome. The criteria for noninferiority required that the upper limits of the 95% confidence intervals were below prespecified margins for both the relative risk (<1.80) and the risk difference (<3.5 percentage points).
Pre-definition of the non-inferiority margin is important. Sometimes a company tests a new drug, hoping to prove superiority over an existing treatment. Superiority is a higher bar than non-inferiority, and superiority trials are more difficult to carry out. But when they work, they give drug companies something to crow about in drug ads and doctors' offices. "Our treatment works better than drug x," they can say.
Many times, drugs just don't meet the goal of superiority. So researchers may then be tempted to go back and try to find non-inferiority in their results as a consolation prize. Non-inferiority, especially if it shows other advantages like fewer side effects, may save the prospects of a drug that didn't turn out to work as well as researchers had hoped.
But if a non-inferiority margin hasn't been pre-defined, it would be tempting for investigators—knowing what they already do about the study results, to come up with a biased non-inferiority margin. It's only appropriate to switch from superiority to non-inferiority if the non-inferiority margin was selected before the start of the study. You can usually find details of the original design of the study by checking clinicaltrials.gov.
By the same token, if a drug performed significantly better than expected in a non-inferiority trial, investigators can claim superiority if they first ran the numbers to show inferiority, because that doesn't bias their results.
That is what happened in the Apixaban trial. The drug met both tests for non-inferiority in its primary outcome, prevention of recurrent venous thromboembolism. It also passed superiority tests for an important secondary outcome, major bleeding events. Patients taking Apixaban had less major bleeding than those taking standard therapy.
Here are a few other cautions about non-inferiority trials. Trials that go for flat-out superiority are tougher to conduct and, if they are sloppy – randomization isn't well done or people don't take their medications as directed, for example – drugs are more likely to fail a test of superiority. But the opposite is true for non-inferiority. Sloppiness in research can make non-inferiority more likely.
On that point, there are two important things to pick out of a non-inferiority trial – the fates of the per-protocol population (PP) and the intent-to-treat population (ITT). If the ITT group shows non-inferiority, but the PP doesn't, that should raise an eyebrow, and frequently does for regulators. Because that means that the people who were most closely following the study guidelines, for example, they consistently took their medicine and took for it for as long as they were supposed to, weren't being helped as much by the new drug as the old one.
And check the dose of the older comparison drug. If it's lower than what doctors might normally prescribe for a given disease, that can skew the study toward the tested drug.
Comparison drugs can also be subject to a phenomenon called biocreep. Basically, biocreep can happen when successive drugs enter the market, each by demonstrating non-inferiority to the drug that came just before. If those non-inferiority margins are wide, it may mean that the tested drug really isn't as good as older treatments on the market because it was tested against the latest in a succession of medications that were each slightly less effective than older tried-and-true drugs. That's one reason regulatory agencies advise drug manufacturers to test new drugs against gold-standard therapies and not other new drugs.