Are most scientific studies screwed up?

[Insp. Clue!So?]

http://volokh.com/posts/1189863895.shtml

From the blog:

[Jonathan Adler, September 15, 2007 at 9:44am] Trackbacks
Are Most Scientific Studies Sloppy?

This week's "Science Journal" column in the W$J (link for subscribers) reports on the interesting and provocative research of John Ioannidis, who argues that the results of most published scientific studies are wrong.

Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true." . . .

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.

He further argues that only a fraction of incorrect studies are ever corrected or retracted, meaning that there are many published studies still "on the books" that support erroneous findings. It's an interesting and provocative thesis.

[luckyme]

[ QUOTE ]
John Ioannidis, who argues that the results of most published scientific studies are wrong.

[/ QUOTE ]
[ QUOTE ]
"There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims,"

[/ QUOTE ]

Anyone care to connect those two for me?
I can understand the conclusions being 'wrong' or poorly supported or the claim overstated.
I'm not following how the study can be wrong, unless he means faking the data, in which case how could he know that just by looking at it.

His challenge of the studies seems to be doing exactly what peer review is supposed to do... check, challenge, confirm.

Does this mean the next mars mission will land on mercury because " in modern research, false findings.." and that Pluto is a planet? or what.

Or has he overgeneralized from his field of study, epidemiology, which may suffer from it's own limitations, to make a sweeping statement about scientific research in general.
Iow, is his study overreaching it's conclusion also?

luckyme

[Insp. Clue!So?]

I think he may be on firmer ground than you suggest. Skeptical Inquirer published an article some years back showing how poorly many studies handled "P" values and statistical significance. I'll try to dig it up.

[tolbiny]

If his study is correct, then its highly likely that his study is incorrect, and if its wrong then that increases the odds that its right!

Catch 22. How can we trust a scientific study pointing out how easy it is for poor science to make it to publication?

[Splendour]

The author of that article said that science isn't infallible in very polite terms...Science research today always seems to be trying to prove things way too fast...If the scientists would just follow the old adage "Time Will Tell" in their work a little bit we wouldn't be at the mercy of scientific errors so much...People are fallible, people are weakened by their own ego problems and are more frequently than not driven by a profit motive...Can't think of anything more at odds with science than profit...
The FDA in recent history had more drug recalls in the last few years than in all of the preceding decades put together...why?..they were rushing to get drugs on to the market and the drugs had lots of side effects... Seems science doesn't know as much as it does know....

[luckyme]

[ QUOTE ]
The FDA in recent history had more drug recalls in the last few years than in all of the preceding decades put together...why?

[/ QUOTE ]

not only that, but the number of airplane crashes has gone through the roof in the last 100 years.

luckyme

[vhawk01]

[ QUOTE ]
[ QUOTE ]
The FDA in recent history had more drug recalls in the last few years than in all of the preceding decades put together...why?

[/ QUOTE ]

not only that, but the number of airplane crashes has gone through the roof in the last 100 years.

luckyme

[/ QUOTE ]

Yeah...wow.

[Iconoclastic]

In my estimation 100% of scientific studies mentioned on Yahoo's front page are screwed up.

[vhawk01]

[ QUOTE ]
In my estimation 100% of scientific studies mentioned on Yahoo's front page are screwed up.

[/ QUOTE ]


They probably all are. Its nearly impossible to make a flawless study. Thats why God invented the phrase "fatally flawed."

[Phil153]

This is very true, but mostly in the soft "sciences". The more papers I read from that field, the more I realize that it's a sad joke. Apart from screwed up methodology, so many conclusions are reached that step way beyond the bounds of the data and fail to look at other explanations/scenarios.

Most of the hard sciences, at least where based on data, are quite reliable IMO.