Lies, damn lies and sta­tis­tics: A Stan­ford wiz says P<0.05 of­fers de­cep­tive ev­i­dence of bio­phar­mas' drug claims

The bio­phar­ma R&D world re­volves around one sim­ple for­mu­la: A P val­ue of less than 0.05 in a piv­otal study. But a top pro­fes­sor of med­i­cine and sta­tis­tics at Stan­ford says it’s a poor mea­sure of val­ue, and he wants to scrap it for some­thing far more de­mand­ing — and far more valu­able.

Writ­ing in the Jour­nal of the Amer­i­can Med­ical As­so­ci­a­tion, John P A Ioan­ni­dis notes that the P val­ue cut­off “is wrong­ly equat­ed with a find­ing or an out­come…be­ing true, valid, and worth act­ing on. These mis­con­cep­tions af­fect re­searchers, jour­nals, read­ers, and users of re­search ar­ti­cles, and even me­dia and the pub­lic who con­sume sci­en­tif­ic in­for­ma­tion.”

And they’re of­ten sim­ply wrong.

Most claims sup­port­ed with P val­ues slight­ly be­low .05 are prob­a­bly false (ie, the claimed as­so­ci­a­tions and treat­ment ef­fects do not ex­ist). Even among those claims that are true, few are worth act­ing on in med­i­cine and health care.

There are just too many ways to game the clin­i­cal tri­al sys­tem, Ioan­ni­dis adds. By fo­cus­ing on small­er ben­e­fits and risks, he writes, you boost the risk that bi­as­es will have an af­fect.

“Mov­ing the P val­ue thresh­old from .05 to .005 will shift about one-third of the sta­tis­ti­cal­ly sig­nif­i­cant re­sults of past bio­med­ical lit­er­a­ture to the cat­e­go­ry of just ‘sug­ges­tive.’ This shift is es­sen­tial for those who be­lieve (per­haps crude­ly) in black and white, sig­nif­i­cant or non­signif­i­cant cat­e­go­riza­tions.”

Ioan­ni­dis, though, is quick to as­sert that there are no easy so­lu­tions to the P val­ue co­nun­drum. There are ad­van­tages, and some big dis­ad­van­tages, for do­ing away with the old stan­dard that can’t be ig­nored.

Adopt­ing low­er P val­ue thresh­olds may help pro­mote a re­formed re­search agen­da with few­er, larg­er, and more care­ful­ly con­ceived and de­signed stud­ies with suf­fi­cient pow­er to pass these more de­mand­ing thresh­olds. How­ev­er, col­lat­er­al harms may al­so emerge. Bias may es­ca­late rather than de­crease if re­searchers and oth­er in­ter­est­ed par­ties (eg, for-prof­it spon­sors) try to find ways to make the re­sults have low­er P val­ues. Se­lect­ed study end­points may be­come even less clin­i­cal­ly rel­e­vant be­cause it is eas­i­er to reach low­er P val­ues with weak sur­ro­gate end points than with hard clin­i­cal out­comes. More­over, re­sults that pass a low­er P val­ue thresh­old may be lim­it­ed by greater re­gres­sion to the mean and new dis­cov­er­ies may have even more ex­ag­ger­at­ed ef­fect sizes than be­fore.

My bet is that the in­dus­try has be­come so fo­cused on beat­ing 0.05, no one will want to drop it for an untest­ed ap­proach that could throw the whole $160 bil­lion drug de­vel­op­ment busi­ness in­to a tizzy. There are no sim­ple bound­ary lines be­tween good and bad. But it’s def­i­nite­ly worth keep­ing in mind the next time you see a bio­phar­ma com­pa­ny cel­e­brat­ing a P val­ue in the 0.04 range of things.


Im­age: John P. A. Ioan­ni­dis. Eras­mus MC via YOUTUBE

5AM Ven­tures: Fu­el­ing the Next Gen­er­a­tion of In­no­va­tors

By RBC Capital Markets
With Andy Schwab, Co-Founder and Managing Partner at 5AM Ventures

Key Points

Prescription Digital Therapeutics, cell therapy technologies, and in silico medicines will be a vital part of future treatment modalities.
Unlocking the potential of the microbiome could be the missing link to better disease diagnosis.
Growing links between academia, industry, and venture capital are spinning out more innovative biotech companies.
Biotech is now seen by investors as a growth space as well as a safe haven, fuelling the recent IPO boom.

Biohaven CEO Vlad Coric (Photo Credit: Andrew Venditti)

Pssst: That big Bio­haven Alzheimer's study? It was a bust. Even the sub­group analy­sis ex­ecs tout­ed was a flop

You know it’s bad when a biopharma player plucks out a subgroup analysis for a positive take — even though it was way off the statistical mark for success, like everything else.

So it was for Biohaven $BHVN on MLK Monday, as the biotech reported on the holiday that their Phase II/III Alzheimer’s study for troriluzole flunked both co-primary endpoints as well as a key biomarker analysis.

The drug — a revised version of the ALS drug riluzole designed to regulate glutamate — did not “statistically differentiate” from placebo on the Alzheimer’s Disease Assessment Scale-Cognitive Subscale 11 (ADAS-cog) and the Clinical Dementia Rating Scale Sum of Boxes (CDR-SB).  The “hippocampal volume” assessment by MRI also failed to distinguish itself from placebo for all patients fitting the mild-to-moderate disease profile they had established for the study.

Endpoints Premium

Premium subscription required

Unlock this article along with other benefits by subscribing to one of our paid plans.

Janet Woodcock (AP Images)

Janet Wood­cock to be act­ing FDA com­mis­sion­er while Biden team fi­nal­izes nom­i­nee — re­ports

Janet Woodcock is set to be the most powerful person at the FDA in less than a week.

The veteran regulator and longtime director of the Center for Drug Evaluation and Research has been tapped as acting commissioner of the FDA, according to reports by BioCentury’s Steve Usdin and Pink Sheet’s Sarah Karlin-Smith.

The appointment was requested by the incoming Biden team, Karlin-Smith added, as they sort out the nomination of a permanent successor to Stephen Hahn — whose one-year tenure has been defined by Covid-19.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.

Janet Woodcock (AP Images)

Janet Wood­cock is in the run­ning for FDA com­mis­sion­er — what does that mean for the agen­cy's fu­ture?

Just a day after reports emerged that Janet Woodcock will serve as interim chief of the FDA, word has gotten out that she is also in the running for the permanent job.

The decision, as the initial wave of reactions suggest, could have dramatic implications for where the agency is headed in the next four years — if not beyond.

Woodcock, the longtime CDER director, is being vetted alongside former FDA principal deputy commissioner Joshua Sharfstein, Bloomberg reported. Already tapped as acting head of the agency, she’s set to take over from Stephen Hahn right after Biden’s inauguration next week.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.

Steve Harr (L) and Hans Bishop

Paint­ing by the num­bers, Sana founders carve up a gi­ant uni­corn-sized IPO — for a biotech that has­n't quite made it to the clin­ic

Sana Biotechnology is one of those startups that was sketched in on the chalkboard day one in the shape of a unicorn.

A giant unicorn.

And from the numbers the cell therapy 2.0 play spelled out in their S-1 $SANA, it’s clear that the company founders — led by a pair of major VCs aligned with some high-profile industry figures — are hunting a big chunk of that value for themselves.

The raise they penciled in — $150 million — isn’t likely what they actually have in mind, and it doesn’t do justice to the size of their ambitions.

Endpoints Premium

Premium subscription required

Unlock this article along with other benefits by subscribing to one of our paid plans.

CEO Brett Monia (Ionis)

Can Brett Mo­nia push Io­n­is be­yond Spin­raza?

For 30 years, Brett Monia struggled as one of Ionis’ top scientists to get their antisense technology to work. Now, as CEO, he’s trying to use it to turn Ionis into one of the industry’s biggest biotechs.

Monia, one of the handful of young scientists who in 1989 followed Stanley Crooke across the country from SmithKline (now GSK) in Philadelphia to found Ionis in Northern California, replaced Crooke as CEO last January. By then, they had proven antisense, an RNA-based method for manipulating gene expression, could work dramatically well in at least some instances, transforming spinal muscular atrophy with the Biogen-partnered blockbuster Spinraza.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.

Terry Rosen, Arcus CEO

Gilead part­ner Ar­cus earns an­a­lyst­s' plau­dits for ear­ly pan­cre­at­ic can­cer da­ta that 'ex­ceed­ed ex­pec­ta­tion­s'

Arcus’ small molecule CD73 inhibitor for pancreatic cancer got a standing ovation from analysts who said preliminary data “exceeded expectations”— making waves in a field that’s seen little progress in several years and proving the candidate could be worth the hundreds of millions Gilead provided upfront in a deal that included more than a billion dollars for opt-in rights and milestones.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.

David Kessler in April 2009 (Eric Risberg/AP Images)

Covid-19 roundup: Hack­ers start re­leas­ing 'ma­nip­u­lat­ed' Covid-19 vac­cine docs; Ex-FDA com­mish David Kessler to re­place Mon­cef Slaoui as Op­er­a­tion Warp Speed chief — re­port

There’s a new twist on the EMA Covid-19 hacking story.

Friday the European agency put out the 5th in a series of statements about the hackers who broke into their system, noting that some of the information on vaccines that was gleaned in the attack is showing up online — altered to raise questions about the Covid-19 vaccines now in use.

This included internal/confidential email correspondence dating from November, relating to evaluation processes for COVID-19 vaccines. Some of the correspondence has been manipulated by the perpetrators prior to publication in a way which could undermine trust in vaccines.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.

With KRAS break­through on the hori­zon, Am­gen's David Reese re­flects on so­tora­si­b's loom­ing re­view date and murky fu­ture

After decades of failures laid waste to R&D outfits looking to solve the KRAS G12C puzzle, Amgen is as close as anyone ever has been to an approval with sotorasib. For Amgen R&D head David Reese, the drug’s looming review date is a point of reflection for his own career and a big milestone for Amgen’s blooming — if controversial — next-gen oncology pipeline.

Amgen filed its FDA application for sotorasib in December to treat metastatic non-small cell lung cancer with the KRAS mutation — once thought to be “undruggable” — months after the agency offered its breakthrough designation based on pivotal Phase I data showing previously unheard of response rates.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 98,100+ biopharma pros reading Endpoints daily — and it's free.