Lies, damn lies and sta­tis­tics: A Stan­ford wiz says P<0.05 of­fers de­cep­tive ev­i­dence of bio­phar­mas' drug claims

The bio­phar­ma R&D world re­volves around one sim­ple for­mu­la: A P val­ue of less than 0.05 in a piv­otal study. But a top pro­fes­sor of med­i­cine and sta­tis­tics at Stan­ford says it’s a poor mea­sure of val­ue, and he wants to scrap it for some­thing far more de­mand­ing — and far more valu­able.

Writ­ing in the Jour­nal of the Amer­i­can Med­ical As­so­ci­a­tion, John P A Ioan­ni­dis notes that the P val­ue cut­off “is wrong­ly equat­ed with a find­ing or an out­come…be­ing true, valid, and worth act­ing on. These mis­con­cep­tions af­fect re­searchers, jour­nals, read­ers, and users of re­search ar­ti­cles, and even me­dia and the pub­lic who con­sume sci­en­tif­ic in­for­ma­tion.”

And they’re of­ten sim­ply wrong.

Most claims sup­port­ed with P val­ues slight­ly be­low .05 are prob­a­bly false (ie, the claimed as­so­ci­a­tions and treat­ment ef­fects do not ex­ist). Even among those claims that are true, few are worth act­ing on in med­i­cine and health care.

There are just too many ways to game the clin­i­cal tri­al sys­tem, Ioan­ni­dis adds. By fo­cus­ing on small­er ben­e­fits and risks, he writes, you boost the risk that bi­as­es will have an af­fect.

“Mov­ing the P val­ue thresh­old from .05 to .005 will shift about one-third of the sta­tis­ti­cal­ly sig­nif­i­cant re­sults of past bio­med­ical lit­er­a­ture to the cat­e­go­ry of just ‘sug­ges­tive.’ This shift is es­sen­tial for those who be­lieve (per­haps crude­ly) in black and white, sig­nif­i­cant or non­signif­i­cant cat­e­go­riza­tions.”

Ioan­ni­dis, though, is quick to as­sert that there are no easy so­lu­tions to the P val­ue co­nun­drum. There are ad­van­tages, and some big dis­ad­van­tages, for do­ing away with the old stan­dard that can’t be ig­nored.

Adopt­ing low­er P val­ue thresh­olds may help pro­mote a re­formed re­search agen­da with few­er, larg­er, and more care­ful­ly con­ceived and de­signed stud­ies with suf­fi­cient pow­er to pass these more de­mand­ing thresh­olds. How­ev­er, col­lat­er­al harms may al­so emerge. Bias may es­ca­late rather than de­crease if re­searchers and oth­er in­ter­est­ed par­ties (eg, for-prof­it spon­sors) try to find ways to make the re­sults have low­er P val­ues. Se­lect­ed study end­points may be­come even less clin­i­cal­ly rel­e­vant be­cause it is eas­i­er to reach low­er P val­ues with weak sur­ro­gate end points than with hard clin­i­cal out­comes. More­over, re­sults that pass a low­er P val­ue thresh­old may be lim­it­ed by greater re­gres­sion to the mean and new dis­cov­er­ies may have even more ex­ag­ger­at­ed ef­fect sizes than be­fore.

My bet is that the in­dus­try has be­come so fo­cused on beat­ing 0.05, no one will want to drop it for an untest­ed ap­proach that could throw the whole $160 bil­lion drug de­vel­op­ment busi­ness in­to a tizzy. There are no sim­ple bound­ary lines be­tween good and bad. But it’s def­i­nite­ly worth keep­ing in mind the next time you see a bio­phar­ma com­pa­ny cel­e­brat­ing a P val­ue in the 0.04 range of things.


Im­age: John P. A. Ioan­ni­dis. Eras­mus MC via YOUTUBE

At the In­flec­tion Point for the Next Gen­er­a­tion of Can­cer Im­munother­a­py

While oncology researchers have long pursued the potential of cellular immunotherapies for the treatment of cancer, it was unclear whether these therapies would ever reach patients due to the complexity of manufacturing and costs of development. Fortunately, the recent successful development and regulatory approval of chimeric antigen receptor-engineered T (CAR-T) cells have demonstrated the significant benefit of these therapies to patients.

Stéphane Bancel, Moderna CEO

'This is not go­ing to be good': Mod­er­na CEO Ban­cel warns of a 'ma­te­r­i­al drop' in vac­cine ef­fi­ca­cy as Omi­cron spreads

Even as public health officials remain guarded about their comments on the likelihood Omicron will escape the reach of the currently approved Covid-19 vaccines, there’s growing scientific consensus that we’re facing a variant that threatens to overwhelm the vaccine barricades that have been erected.

Stéphane Bancel, the CEO of Moderna, one of the leading mRNA players whose quick vault into the markets with a highly effective vaccine created an instant multibillion-dollar market, added his voice to the rising chorus early Tuesday. According to Bancel, there will be a significant drop in efficacy when the average immune system is confronted by Omicron. The only question now is: How much?

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 124,200+ biopharma pros reading Endpoints daily — and it's free.

Ap­peals court puts the fi­nal nail in the cof­fin for Tec­fidera patent, adding to Bio­gen's bur­geon­ing set­backs

In another setback for Biogen, the big biotech lost its appeal to revive a patent for the once-blockbuster drug Tecfidera, marking a likely conclusion to the case.

The US Court of Appeals for the Federal Circuit issued the ruling Tuesday morning, saying Biogen failed to satisfy the “written description” requirement for patent law. As a result, Mylan-turned-Viatris will be able to sell its multiple sclerosis generic without fear of infringement and Biogen will have to find a new revenue driver elsewhere.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 124,200+ biopharma pros reading Endpoints daily — and it's free.

Philip Dormitzer, new GSK global head of vaccines R&D

Glax­o­SmithK­line poach­es Pfiz­er's vi­ral vac­cines lead in rush to cap­i­tal­ize on fu­ture of mR­NA

GlaxoSmithKline has appointed Philip Dormitzer, formerly chief scientific officer of Pfizer’s viral vaccines unit, as its newest global head of vaccines R&D, looking to leverage one of the leading minds behind Pfizer and BioNTech’s RNA collaboration that led to Covid-19 jab Comirnaty, the British drug giant said Tuesday.

Dormitzer had been with Pfizer for a little more than six years, joining up after a seven-year stint with Novartis, where he reached the role of US head of research and head of global virology for the company’s vaccines and diagnostics unit.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 124,200+ biopharma pros reading Endpoints daily — and it's free.

In­tro­duc­ing End­points Stu­dio, a new way to ad­ver­tise with End­points-craft­ed brand­ing cam­paigns

Since our start in 2016, Endpoints has grown fast while executing our mission to cover biopharma’s most critical developments for industry pros worldwide. As readership has grown, our advertising business has too. Endpoints advertising partners support the mission and engage their desired audiences through announcements on our email and web platforms, brand recognition in our event coverage and sponsorships of Endpoints daily and weekly reports.

As lead drug runs in­to a wall, De­ci­phera slims down its pipeline, puts 140 jobs on the chop­ping block

Barely a month after disappointing data shattered hopes for a major label expansion for the GI tumor drug Qinlock, Deciphera is making a major pivot — scrapping development plans for that drug and discarding another while it hunkers down and focuses on two remaining drugs in the pipeline.

As a result, 140 of its staffers will be laid off.

The restructuring, which claims the equivalent of 35% of its total workforce, will take place across all departments including commercial, R&D as well as general and administrative support functions, Deciphera said, as it looks to streamline Qinlock-related commercial operations in the US while concentrating only on a “select number of key European markets.”

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 124,200+ biopharma pros reading Endpoints daily — and it's free.

How to use reg­istry da­ta to sup­port FDA de­ci­sion mak­ing: Agency ex­plains in new guid­ance

Drugmakers looking to design a new registry or use an existing one to support a regulatory decision on a drug’s effectiveness or safety will need to consult with a new draft guidance released Monday by the FDA.

The agency’s reliance on registry data for regulatory decisions dates back more than two decades, at least, as in 1998 Bayer won approval for its anticoagulant Refludan (withdrawn from the market in 2013 for commercial reasons) based in part on a historical control group pulled from a registry.

Tillman Gerngross (Adagio)

Till­man Gern­gross on Omi­cron: 'It is a grim sit­u­a­tion...we’re go­ing to see a sig­nif­i­cant drop in vac­cine ef­fi­ca­cy'

Tillman Gerngross, the rarely shy Dartmouth professor, biotech entrepreneur and antibody expert, has been warning for over a year that the virus behind Covid-19 would likely continue to mutate, potentially in ways that avoid immunity from infection and the best defenses scientists developed. He spun out a company, Adagio, to build a universal antibody, one that could snuff out any potential mutation.

Endpoints Premium

Premium subscription required

Unlock this article along with other benefits by subscribing to one of our paid plans.

In­cor­po­rat­ing Ex­ter­nal Da­ta in­to Clin­i­cal Tri­als: Com­par­ing Dig­i­tal Twins to Ex­ter­nal Con­trol Arms

Most drug development professionals are familiar with the nerve-racking wait for the read-out of a large trial. If it’s negative, is the investigational therapy ineffective? Or could the failure result from an unforeseen flaw in the design or execution of the protocol, rather than a lack of efficacy? The team could spend weeks analyzing data, but a definitive answer may be elusive due to insufficient power for such analyses in the already completed trial. These problems are only made worse if the trial had lower enrollment, or higher dropout than expected due to an unanticipated event like COVID-19. And if a trial is negative, the next one is likely to be larger and more costly — if it happens at all.