Lies, damn lies and sta­tis­tics: A Stan­ford wiz says P<0.05 of­fers de­cep­tive ev­i­dence of bio­phar­mas' drug claims

The bio­phar­ma R&D world re­volves around one sim­ple for­mu­la: A P val­ue of less than 0.05 in a piv­otal study. But a top pro­fes­sor of med­i­cine and sta­tis­tics at Stan­ford says it’s a poor mea­sure of val­ue, and he wants to scrap it for some­thing far more de­mand­ing — and far more valu­able.

Writ­ing in the Jour­nal of the Amer­i­can Med­ical As­so­ci­a­tion, John P A Ioan­ni­dis notes that the P val­ue cut­off “is wrong­ly equat­ed with a find­ing or an out­come…be­ing true, valid, and worth act­ing on. These mis­con­cep­tions af­fect re­searchers, jour­nals, read­ers, and users of re­search ar­ti­cles, and even me­dia and the pub­lic who con­sume sci­en­tif­ic in­for­ma­tion.”

And they’re of­ten sim­ply wrong.

Most claims sup­port­ed with P val­ues slight­ly be­low .05 are prob­a­bly false (ie, the claimed as­so­ci­a­tions and treat­ment ef­fects do not ex­ist). Even among those claims that are true, few are worth act­ing on in med­i­cine and health care.

There are just too many ways to game the clin­i­cal tri­al sys­tem, Ioan­ni­dis adds. By fo­cus­ing on small­er ben­e­fits and risks, he writes, you boost the risk that bi­as­es will have an af­fect.

“Mov­ing the P val­ue thresh­old from .05 to .005 will shift about one-third of the sta­tis­ti­cal­ly sig­nif­i­cant re­sults of past bio­med­ical lit­er­a­ture to the cat­e­go­ry of just ‘sug­ges­tive.’ This shift is es­sen­tial for those who be­lieve (per­haps crude­ly) in black and white, sig­nif­i­cant or non­signif­i­cant cat­e­go­riza­tions.”

Ioan­ni­dis, though, is quick to as­sert that there are no easy so­lu­tions to the P val­ue co­nun­drum. There are ad­van­tages, and some big dis­ad­van­tages, for do­ing away with the old stan­dard that can’t be ig­nored.

Adopt­ing low­er P val­ue thresh­olds may help pro­mote a re­formed re­search agen­da with few­er, larg­er, and more care­ful­ly con­ceived and de­signed stud­ies with suf­fi­cient pow­er to pass these more de­mand­ing thresh­olds. How­ev­er, col­lat­er­al harms may al­so emerge. Bias may es­ca­late rather than de­crease if re­searchers and oth­er in­ter­est­ed par­ties (eg, for-prof­it spon­sors) try to find ways to make the re­sults have low­er P val­ues. Se­lect­ed study end­points may be­come even less clin­i­cal­ly rel­e­vant be­cause it is eas­i­er to reach low­er P val­ues with weak sur­ro­gate end points than with hard clin­i­cal out­comes. More­over, re­sults that pass a low­er P val­ue thresh­old may be lim­it­ed by greater re­gres­sion to the mean and new dis­cov­er­ies may have even more ex­ag­ger­at­ed ef­fect sizes than be­fore.

My bet is that the in­dus­try has be­come so fo­cused on beat­ing 0.05, no one will want to drop it for an untest­ed ap­proach that could throw the whole $160 bil­lion drug de­vel­op­ment busi­ness in­to a tizzy. There are no sim­ple bound­ary lines be­tween good and bad. But it’s def­i­nite­ly worth keep­ing in mind the next time you see a bio­phar­ma com­pa­ny cel­e­brat­ing a P val­ue in the 0.04 range of things.


Im­age: John P. A. Ioan­ni­dis. Eras­mus MC via YOUTUBE

Op­ti­miz­ing Cell and Gene Ther­a­py De­vel­op­ment and Pro­duc­tion: How Tech­nol­o­gy Providers Like Corn­ing Life Sci­ences are Spurring In­no­va­tion

Remarkable advances in cell and gene therapy over the last decade offer unprecedented therapeutic promise and bring new hope for many patients facing diseases once thought incurable. However, for cell and gene therapies to reach their full potential, researchers, manufacturers, life science companies, and academics will need to work together to solve the significant challenges facing the industry.

Pre­sent­ing a live End­points News event: Man­ag­ing a biotech in tur­bu­lent times

Biotech is one of the smartest, best educated industries on the planet. PhDs abound. We’ve had a long enough track record to see a new generation of savvy, experienced execs coming together to run startups.

And in these times, they are being tested as never before.

Biotech is going through quite a rough patch right now. For 2 years, practically anyone with a decent resume and some half-baked ideas on biotech could start a company and get it funded. The pandemic made it easy in many ways to pull off an IPO, with traditional road shows shut down in exchange for a series of quick Zoom meetings. Generalist investors flocked as the numbers raised soared into the stratosphere.

David Baker working with a student on their protein design (Jason Mast)

Sci­en­tists are fi­nal­ly learn­ing how to de­sign pro­teins from scratch. Drug de­vel­op­ment may nev­er be the same

SEATTLE — It’s a cloudy Thursday afternoon in mid-July and David Baker is reclining into the futon in his corner office at the University of Washington, arms splayed out like a daytime talk show host as he coaches another one of his postdocs through the slings and arrows of scientific celebrity.

“Be jealous of your time,” he says, before plotting ways of sneaking her out of Zooms. “It’s this horrible cost to science that you’re tied up in some stupid meeting.”

Endpoints Premium

Premium subscription required

Unlock this article along with other benefits by subscribing to one of our paid plans.

Amidst R&D reshuf­fle, Ver­tex ex­pands its pres­ence in Boston, aim­ing to be­come num­ber one

Vertex Pharmaceuticals has been one of the buzzier names in the bustling Boston biotech scene, but now the company is looking to vault to number one status — at least in terms of physical footprint.

At a ribbon cutting on Tuesday for its new Jeffrey Leiden Center for Cell and Genetic Therapies at the Boston Seaport, Vertex announced it would embark on a new project: The company will build a 344,000 square foot facility in the seaport to accommodate the company’s growing R&D needs, especially in its cell and gene therapies program.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 142,200+ biopharma pros reading Endpoints daily — and it's free.

Patty Murray, D-WA (Graeme Sloan/Sipa USA)(Sipa via AP Images)

Sen­ate user fee reau­tho­riza­tion bill omits ac­cel­er­at­ed ap­proval re­forms, shows wide gaps with House ver­sion

The Senate health committee on Tuesday released its first version of the bill to reauthorize all the different FDA user fees. But unlike the House version, there are only a few controversial items in the Senate’s version, which does not address either accelerated approval reforms or clinical trial diversity (as the House did).

While it’s still relatively early in the process of finalizing this legislation (the ultimate statutory deadline is the end of September), the House and Senate, at least initially, appear to be starting off in different corners on what should be included.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 142,200+ biopharma pros reading Endpoints daily — and it's free.

Warren Buffett, Berkshire Hathaway CEO

Berk­shire Hath­away pulls out of Ab­b­Vie, Bris­tol My­ers Squibb in­vest­ments

It looks like Warren Buffett is sticking to ice cream and railroads for the moment.

The billionaire CEO of Berkshire Hathaway backed out of two major holdings in the pharma industry, Forexlive first reported, including a $410 million investment in AbbVie and a $324.4 million stake in Bristol Myers Squibb.

The move comes after Berkshire abandoned its Teva shares just last quarter, Bloomberg reported.

Long-ex­pect­ed UK lay­offs im­mi­nent for No­var­tis fol­low­ing sale

Nearly a year ago, more than 200 workers at Novartis’ Grimsby, UK, facility were able to hang on to their jobs after the pharma closed a Switzerland site as a part of its workforce restructuring plan. Now, it looks like those employees’ time is up, as the site has been sold, Grimsby Telegraph reported today.

The manufacturing site has been sold to Humber Industrials, a subsidiary of International Process Plants. None of the current staff members will be working with the new owners, however.

Clay Siegall (Photo by Dimitrios Kambouris/Getty Images for Gabrielle's Angel Foundation)

UP­DAT­ED: Clay Sie­gall re­signs from Seagen amid in­ves­ti­ga­tion in­to do­mes­tic vi­o­lence claims

A week after Seagen revealed that longtime CEO Clay Siegall was on leave due to an allegation of domestic violence, he has resigned.

Since that shocking revelation, more details about the claims have emerged into the public eye. As Endpoints News reported, Siegall was arrested on April 23. A police report about that night and a subsequent temporary restraining order described a pattern of abusive behavior against his wife and a physical altercation that left her with multiple bruises. Siegall denied the claims.

Endpoints News

Keep reading Endpoints with a free subscription

Unlock this story instantly and join 142,200+ biopharma pros reading Endpoints daily — and it's free.

FDA lob­bies Con­gress over rare dis­ease court rul­ing with wide im­pli­ca­tions

Usually reserved for making decisions on drug applications or enforcing what Congress stipulates, the FDA is now dipping its toe into the wild world of congressional politics as it attempts to fix a major court decision that could have a chilling effect on rare disease R&D.

The case in question from last October saw a US appeals court overturn a prior FDA court win, saying that the agency never should’ve approved a rare disease drug because a previously approved but more expensive drug with the same active ingredient has orphan drug exclusivity barring such an approval.