Under the subject line, “another crappy study,” Jay Brophy writes:
I don’t know if you will have time for this as it’s about another crappy study:
What is especially irritating is the imprimatur provided by publication in a Nature journal and of using a highly recognizable dataset which has historically been analyzed with much rigour and care.
This problem of poor secondary analyses of “respected” datasets is likely to be an increasing problem.
Without going into too much detail the paper suffers from measurement error, selection bias, missing data, inferential errors, researcher degrees of freedom, model mis-specification, poor reporting, and elementary data analytical errors.
For example in the unadjusted model they report Q2:Q1 1.03 (0.96, 1.10) but a quick look at Table says the Q2:Q1 is actually 0.98 (0.92, 1.05).
Maybe I shouldn’t expect much when I read “The data were generated by GraphPad Prism 9.4 and R version 3.6”.
I guess my question is how bad must it be before retraction becomes appropriate?
I don’t have the energy to go through the paper in question, so I’ll just answer the last question, about retraction. My quick answer is that retraction is not scalable: the amount of effort required to secure a retraction is just too much.
To put it another way: Yeah, I’m guessing that retraction would appropriate in the above case, just as it would be appropriate to retract the paper discussed here, just as it would probably be appropriate to retract some large percentage of papers published in Psychological Science and PNAS during the “hell decade” of 2005-2015 . . . but it’s not going to happen. The American Statistical Association and the American Political Science Association aren’t even going to take back the prestigious awards they gave to prominent plagiarists. Nothings’s gonna happen. The mayor of NYC has not gone to jail, Columbia University falsified its U.S. News numbers and no one paid the price, that data-fudging sleep guy is still a professor at the University of California, as is John Yoo, the cold-shower guy is still teaching at Stanford, the giraffe guy is still at USC, Ted talks keep on happening, . . .
I think we have no power but to scream.
