Lies, damn lies and sta­tis­tics: A Stan­ford wiz says P<0.05 of­fers de­cep­tive ev­i­dence of bio­phar­mas' drug claims

The bio­phar­ma R&D world re­volves around one sim­ple for­mu­la: A P val­ue of less than 0.05 in a piv­otal study. But a top pro­fes­sor of med­i­cine and sta­tis­tics at Stan­ford says it’s a poor mea­sure of val­ue, and he wants to scrap it for some­thing far more de­mand­ing — and far more valu­able.

Writ­ing in the Jour­nal of the Amer­i­can Med­ical As­so­ci­a­tion, John P A Ioan­ni­dis notes that the P val­ue cut­off “is wrong­ly equat­ed with a find­ing or an out­come…be­ing true, valid, and worth act­ing on. These mis­con­cep­tions af­fect re­searchers, jour­nals, read­ers, and users of re­search ar­ti­cles, and even me­dia and the pub­lic who con­sume sci­en­tif­ic in­for­ma­tion.”

And they’re of­ten sim­ply wrong.

Most claims sup­port­ed with P val­ues slight­ly be­low .05 are prob­a­bly false (ie, the claimed as­so­ci­a­tions and treat­ment ef­fects do not ex­ist). Even among those claims that are true, few are worth act­ing on in med­i­cine and health care.

There are just too many ways to game the clin­i­cal tri­al sys­tem, Ioan­ni­dis adds. By fo­cus­ing on small­er ben­e­fits and risks, he writes, you boost the risk that bi­as­es will have an af­fect.

“Mov­ing the P val­ue thresh­old from .05 to .005 will shift about one-third of the sta­tis­ti­cal­ly sig­nif­i­cant re­sults of past bio­med­ical lit­er­a­ture to the cat­e­go­ry of just ‘sug­ges­tive.’ This shift is es­sen­tial for those who be­lieve (per­haps crude­ly) in black and white, sig­nif­i­cant or non­signif­i­cant cat­e­go­riza­tions.”

Ioan­ni­dis, though, is quick to as­sert that there are no easy so­lu­tions to the P val­ue co­nun­drum. There are ad­van­tages, and some big dis­ad­van­tages, for do­ing away with the old stan­dard that can’t be ig­nored.

Adopt­ing low­er P val­ue thresh­olds may help pro­mote a re­formed re­search agen­da with few­er, larg­er, and more care­ful­ly con­ceived and de­signed stud­ies with suf­fi­cient pow­er to pass these more de­mand­ing thresh­olds. How­ev­er, col­lat­er­al harms may al­so emerge. Bias may es­ca­late rather than de­crease if re­searchers and oth­er in­ter­est­ed par­ties (eg, for-prof­it spon­sors) try to find ways to make the re­sults have low­er P val­ues. Se­lect­ed study end­points may be­come even less clin­i­cal­ly rel­e­vant be­cause it is eas­i­er to reach low­er P val­ues with weak sur­ro­gate end points than with hard clin­i­cal out­comes. More­over, re­sults that pass a low­er P val­ue thresh­old may be lim­it­ed by greater re­gres­sion to the mean and new dis­cov­er­ies may have even more ex­ag­ger­at­ed ef­fect sizes than be­fore.

My bet is that the in­dus­try has be­come so fo­cused on beat­ing 0.05, no one will want to drop it for an untest­ed ap­proach that could throw the whole $160 bil­lion drug de­vel­op­ment busi­ness in­to a tizzy. There are no sim­ple bound­ary lines be­tween good and bad. But it’s def­i­nite­ly worth keep­ing in mind the next time you see a bio­phar­ma com­pa­ny cel­e­brat­ing a P val­ue in the 0.04 range of things.


Im­age: John P. A. Ioan­ni­dis. Eras­mus MC via YOUTUBE

Fol­low­ing news of job cuts in Eu­ro­pean R&D ops, Sanofi con­firms it’s of­fer­ing US work­ers an 'ear­ly ex­it'

Ear­li­er in the week we learned that Sanofi was bring­ing out the bud­get ax to trim 466 R&D jobs in Eu­rope, re­tool­ing its ap­proach to car­dio as re­search chief John Reed beefed up their work in can­cer and gene ther­a­pies. And we’re end­ing the week with news that the phar­ma gi­ant has al­so been qui­et­ly re­duc­ing staff in the US, tar­get­ing hun­dreds of jobs as the com­pa­ny push­es vol­un­tary buy­outs with a fo­cus on R&D sup­port ser­vices.

Why would the FDA ap­prove an­oth­er con­tro­ver­sial drug to spur a woman’s li­bido with these da­ta? And why no ex­pert pan­el re­view?

AMAG Pharmaceuticals’ newly approved drug for spurring women’s sexual desire may never make much money, but it’s a big hit at sparking media attention.

The therapy — Vyleesi (bremelanotide) — got the green light from regulators on Friday evening, swiftly lighting up a range of stories around the world, from The New York Times to The Guardian. Several headlines inevitably referred to it as the “female Viagra,” invoking Pfizer’s old erectile dysfunction blockbuster.

But the two drugs have little in common.

Endpoints News

Basic subscription required

Unlock this story instantly and join 53,300+ biopharma pros reading Endpoints daily — and it's free.

Roger Perlmutter. Merck via webcast

'Our lega­cy mat­ter­s': Mer­ck maps out Keytru­da king­dom while spot­light­ing ad­vances in vac­cines, hos­pi­tal care

“You can for the mo­ment stop tak­ing notes. You can put down your pens and your pad. I have no slides. I have no sub­stan­tive da­ta. I have no pitch.”

So be­gan Roger Perl­mut­ter’s brief ap­pear­ance on­stage at Mer­ck’s first in­vestor day in five years, where he dived in­to the com­pa­ny’s his­to­ry dat­ing back to 1933. The first em­ploy­ees at Mer­ck Re­search Lab­o­ra­to­ries, hand­picked by founder George W. Mer­ck, were crit­i­cal to Mer­ck’s abil­i­ty to achieve clin­i­cal and com­mer­cial suc­cess.

How small- to mid-sized biotechs can adopt pa­tient cen­tric­i­ty in their on­col­o­gy tri­als

By Lucy Clos­sick Thom­son, Se­nior Di­rec­tor of On­col­o­gy Pro­ject Man­age­ment, Icon

Clin­i­cal tri­als in on­col­o­gy can be cost­ly and chal­leng­ing to man­age. One fac­tor that could re­duce costs and re­duce bar­ri­ers is har­ness­ing the pa­tient voice in tri­al de­sign to help ac­cel­er­ate pa­tient en­roll­ment. Now is the time to adopt pa­tient-cen­tric strate­gies that not on­ly fo­cus on pa­tient needs, but al­so can main­tain cost ef­fi­cien­cy.

John Reed at JPM 2019. Jeff Rumans for Endpoints News

Sanofi's John Reed con­tin­ues to re­or­ga­nize R&D, cut­ting 466 jobs while boost­ing can­cer, gene ther­a­py re­search

The R&D reorganization inside Sanofi is continuing, more than a year after the pharma giant brought in John Reed to head the research arm of the Paris-based company.

Endpoints News

Basic subscription required

Unlock this story instantly and join 53,300+ biopharma pros reading Endpoints daily — and it's free.

Re­gen­eron/Sanofi's an­ti­body un­der­whelms in asth­ma study — shares of ri­val Anap­tys­Bio pay the price

Al­though ex­pec­ta­tions were mut­ed, Re­gen­eron $REGN and Sanofi’s $SNY ex­per­i­men­tal IL-33 an­ti­body has un­der­whelmed in a proof-of-con­cept mid-stage asth­ma tri­al. Al­though the drug sig­nif­i­cant­ly im­proved the loss of asth­ma con­trol and lung func­tion as a monother­a­py com­pared to a place­bo, its ef­fect was nei­ther su­pe­ri­or to the es­tab­lished Dupix­ent, nor of val­ue when com­bined with the IL-4/IL-13 treat­ment.

Green-light­ed in Japan, FDA quick­ly spurns Dai­ichi Sanky­o's flawed ap­pli­ca­tion for AML drug

Three days af­ter win­ning Japan­ese ap­proval for its acute myeloid leukemia drug quizar­tinib, Dai­ichi Sankyo is be­ing forced to en­dure an em­bar­rass­ing re­jec­tion at the hands of the FDA.

US reg­u­la­tors wast­ed no time in bat­ting back quizar­tinib af­ter first high­light­ing the messy da­ta in its ap­pli­ca­tion in an in­ter­nal re­view, that in turn per­suad­ed a large ma­jor­i­ty of out­side ex­perts to rec­om­mend a re­jec­tion for the drug, which tar­gets FLT3-ITD–pos­i­tive AML cas­es.

Which top 10 big phar­mas have the most to gain — or lose — over the next 5 years?

When Evaluate Pharma crunched the likely drug sales numbers for the big 10, 2 stood out. 

Takeda, with its big Shire buyout under its belt, is set to almost double its worldwide sales record for 2018 over 5 years, putting it in the big 10 — the 9th spot, to be exact — which is exactly where CEO Christophe Weber wants to be. 

Endpoints News

Basic subscription required

Unlock this story instantly and join 53,300+ biopharma pros reading Endpoints daily — and it's free.

HIV, pneu­mo­coc­cal — and what? Mer­ck­'s un­ex­pect­ed pipeline high­light ex­cites a lit­tle biotech

In an R&D update dominated by oncology — mostly Keytruda, followed by Lynparza and Lenvima — Merck chose to highlight a program in sensory pathology, an HIV drug, and a group of pneumococcal vaccines. And that has made at least one biotech very happy.

Endpoints News

Basic subscription required

Unlock this story instantly and join 53,300+ biopharma pros reading Endpoints daily — and it's free.