Lies, damn lies and sta­tis­tics: A Stan­ford wiz says P<0.05 of­fers de­cep­tive ev­i­dence of bio­phar­mas' drug claims

The bio­phar­ma R&D world re­volves around one sim­ple for­mu­la: A P val­ue of less than 0.05 in a piv­otal study. But a top pro­fes­sor of med­i­cine and sta­tis­tics at Stan­ford says it’s a poor mea­sure of val­ue, and he wants to scrap it for some­thing far more de­mand­ing — and far more valu­able.

Writ­ing in the Jour­nal of the Amer­i­can Med­ical As­so­ci­a­tion, John P A Ioan­ni­dis notes that the P val­ue cut­off “is wrong­ly equat­ed with a find­ing or an out­come…be­ing true, valid, and worth act­ing on. These mis­con­cep­tions af­fect re­searchers, jour­nals, read­ers, and users of re­search ar­ti­cles, and even me­dia and the pub­lic who con­sume sci­en­tif­ic in­for­ma­tion.”

And they’re of­ten sim­ply wrong.

Most claims sup­port­ed with P val­ues slight­ly be­low .05 are prob­a­bly false (ie, the claimed as­so­ci­a­tions and treat­ment ef­fects do not ex­ist). Even among those claims that are true, few are worth act­ing on in med­i­cine and health care.

There are just too many ways to game the clin­i­cal tri­al sys­tem, Ioan­ni­dis adds. By fo­cus­ing on small­er ben­e­fits and risks, he writes, you boost the risk that bi­as­es will have an af­fect.

“Mov­ing the P val­ue thresh­old from .05 to .005 will shift about one-third of the sta­tis­ti­cal­ly sig­nif­i­cant re­sults of past bio­med­ical lit­er­a­ture to the cat­e­go­ry of just ‘sug­ges­tive.’ This shift is es­sen­tial for those who be­lieve (per­haps crude­ly) in black and white, sig­nif­i­cant or non­signif­i­cant cat­e­go­riza­tions.”

Ioan­ni­dis, though, is quick to as­sert that there are no easy so­lu­tions to the P val­ue co­nun­drum. There are ad­van­tages, and some big dis­ad­van­tages, for do­ing away with the old stan­dard that can’t be ig­nored.

Adopt­ing low­er P val­ue thresh­olds may help pro­mote a re­formed re­search agen­da with few­er, larg­er, and more care­ful­ly con­ceived and de­signed stud­ies with suf­fi­cient pow­er to pass these more de­mand­ing thresh­olds. How­ev­er, col­lat­er­al harms may al­so emerge. Bias may es­ca­late rather than de­crease if re­searchers and oth­er in­ter­est­ed par­ties (eg, for-prof­it spon­sors) try to find ways to make the re­sults have low­er P val­ues. Se­lect­ed study end­points may be­come even less clin­i­cal­ly rel­e­vant be­cause it is eas­i­er to reach low­er P val­ues with weak sur­ro­gate end points than with hard clin­i­cal out­comes. More­over, re­sults that pass a low­er P val­ue thresh­old may be lim­it­ed by greater re­gres­sion to the mean and new dis­cov­er­ies may have even more ex­ag­ger­at­ed ef­fect sizes than be­fore.

My bet is that the in­dus­try has be­come so fo­cused on beat­ing 0.05, no one will want to drop it for an untest­ed ap­proach that could throw the whole $160 bil­lion drug de­vel­op­ment busi­ness in­to a tizzy. There are no sim­ple bound­ary lines be­tween good and bad. But it’s def­i­nite­ly worth keep­ing in mind the next time you see a bio­phar­ma com­pa­ny cel­e­brat­ing a P val­ue in the 0.04 range of things.


Im­age: John P. A. Ioan­ni­dis. Eras­mus MC via YOUTUBE

A fa­vorite in Alex­ion’s C-suite is leav­ing, and some mighty sur­prised an­a­lysts aren’t the least bit hap­py about it

Analysts hate to lose a biotech CFO they’ve come to trust and admire — especially if they’re being blindsided by a surprise exit.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 59,900+ biopharma pros reading Endpoints daily — and it's free.

David Grainger [file photo]

'Dis­con­nect the bas­tard­s' — one biotech's plan to break can­cer cell­s' uni­fied de­fens­es

Chemotherapy and radiotherapy are the current gladiators of cancer treatment, but they come with well-known limitations and side-effects. The emergence of immunotherapy — a ferocious new titan in oncologist’s toolbox — takes the brakes off the immune system to kill cancer cells with remarkable success in some cases, but the approach is not always effective. What makes certain forms of cancer so resilient? Scientists may have finally pieced together a tantalizing piece of the puzzle, and a new biotech is banking on a new approach to fill the gap.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 59,900+ biopharma pros reading Endpoints daily — and it's free.

While No­var­tis ban­ish­es Zol­gens­ma scan­dal scars — Bio­gen goes on a Spin­raza 'of­fen­sive'

While Novartis painstakingly works to mop up the stench of the data manipulation scandal associated with its expensive gene therapy for spinal muscular atrophy (SMA) Zolgensma— rival Biogen is attempting to expand the use of its SMA therapy, Spinraza. 

The US drugmaker $BIIB secured US approval for Spinraza for use in the often fatal genetic disease in 2016. The approval covered a broad range of patients with infantile-onset (most likely to develop Type 1) SMA. 

Jason Kelly. Mike Blake/Reuters via Adobe

Eye­ing big ther­a­peu­tic push, Gink­go bags $290M to build a cell pro­gram­ming em­pire

Ginkgo Bioworks is on a roll. Days after publicizing a plan to nurture new startups via partnerships with accelerators Y Combinator and Petri, the Boston biotech says it has raised another $290 million for its cell programming platform to reach further and wider.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 59,900+ biopharma pros reading Endpoints daily — and it's free.

UP­DAT­ED: Speak­er Nan­cy Pelosi to un­veil bill for fed­er­al­ly ne­go­ti­at­ed drug prices

After months of buzz from both sides of the aisle, Speaker Nancy Pelosi will today introduce her plan to allow the federal government to negotiate prices for 250 prescription drugs, setting up a showdown with a pharmaceutical industry working overtime to prevent it.

The need to limit drug prices is a rare point of agreement between President Trump and Democrats, although the president has yet to comment on the proposal and will likely face pressure to back a more conservative option or no bill at all. Republican Senator Chuck Grassley is reportedly lobbying his fellow party members on a more modest proposal he negotiated with Democratic Senator Ron Wyden in July.

Jeff Kindler's Cen­trex­ion re­news bid to make pub­lic de­but

Jeffrey Kindler’s plan to take his biotech — which is developing a slate of non-opioid painkillers — public, is back on.

The Boston based company, led by former Pfizer $PFE chief Kindler, originally contemplated a $70 million to $80 million IPO last year— but eventually postponed that strategy. On Wednesday, the company revived its bid to make a public debut in a filing with the SEC — although no pricing details were disclosed.

Zachary Hornby. Boundless

'A fourth rev­o­lu­tion in can­cer ther­a­pies': ARCH-backed Bound­less Bio flash­es big check, makes big­ger promis­es in de­but

It was the cellular equivalent of opening your car door and finding an active, roaring engine in the driver seat.

Scientists learned strands of DNA could occasionally appear outside of its traditional home in the nucleus in the 1970s, when they appeared as little, innocuous circles on microscopes; inexplicable but apparently innate. But not until UC San Diego’s Paul Mischel published his first study in Science in 2014 did researchers realize these circles were not only active but potentially overactive and driving some cancer tumors’ superhuman growth.

It’s fi­nal­ly over: Bio­gen, Ei­sai scrap big Alzheimer’s PhI­I­Is af­ter a pre­dictable BACE cat­a­stro­phe rais­es safe­ty fears

Months after analysts and investors called on Biogen and Eisai to scrap their BACE drug for Alzheimer’s and move on in the wake of a string of late-stage failures and rising safety fears, the partners have called it quits. And they said they were dropping the drug — elenbecestat — after the independent monitoring board raised concerns about…safety.

We don’t know exactly what researchers found in this latest catastrophe, but the companies noted in their release that investigators had determined that the drug was flunking the risk/benefit analysis.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 59,900+ biopharma pros reading Endpoints daily — and it's free.

Mer­ck helps bankroll new part­ner Themis' game plan to fin­ish the chikun­gun­ya race and be­gin on­colyt­ic virus quest

As Themis gears up for a Phase III trial of its chikungunya vaccine, the Vienna-based biotech has closed out €40 million ($44 million) to foot the clinical and manufacturing bills.

Its heavyweight partners at Merck — which signed a pact around a mysterious “blockbuster indication” last month — jumped into the Series D, led by new investors Farallon Capital and Hadean Ventures. Adjuvant Capital also joined, as did current investors Global Health Investment Fund, aws Gruenderfonds, Omnes Capital, Ventech and Wellington Partners Life Sciences.