Are most scientific studies screwed up?

[vhawk01]

[ QUOTE ]
This is very true, but mostly in the soft "sciences". The more papers I read from that field, the more I realize that it's a sad joke. Apart from screwed up methodology, so many conclusions are reached that step way beyond the bounds of the data and fail to look at other explanations/scenarios.

Most of the hard sciences, at least where based on data, are quite reliable IMO.

[/ QUOTE ]

Read more medical journals, they really are pretty awful. In my limited experience anyway.

[tame_deuces]

[ QUOTE ]
This is very true, but mostly in the soft "sciences". The more papers I read from that field, the more I realize that it's a sad joke. Apart from screwed up methodology, so many conclusions are reached that step way beyond the bounds of the data and fail to look at other explanations/scenarios.

Most of the hard sciences, at least where based on data, are quite reliable IMO.

[/ QUOTE ]

In my experience, methodology in soft science (the ones I have read is mostly sociology, psychology) is usually heaps and bounds better applied than in hard science (I've mostly read physics & climatology), and for most soft science papers I have read they have been astutely more aware of their study's limitations than in most hard science papers I have read.

I'll stretch it so far as to say soft science papers due to methodology and the operationalization of variables (since measurability in soft science is often iffy to begin with) must reach less bombastic conclusions, but that is in no way an indicator of less academic provess.

So I politely disagree.

[Insp. Clue!So?]

[ QUOTE ]
[ QUOTE ]
This is very true, but mostly in the soft "sciences". The more papers I read from that field, the more I realize that it's a sad joke. Apart from screwed up methodology, so many conclusions are reached that step way beyond the bounds of the data and fail to look at other explanations/scenarios.

Most of the hard sciences, at least where based on data, are quite reliable IMO.

[/ QUOTE ]

In my experience, methodology in soft science (the ones I have read is mostly sociology, psychology) is usually heaps and bounds better applied than in hard science (I've mostly read physics & climatology), and for most soft science papers I have read they have been astutely more aware of their study's limitations than in most hard science papers I have read.

I'll stretch it so far as to say soft science papers due to methodology and the operationalization of variables (since measurability in soft science is often iffy to begin with) must reach less bombastic conclusions, but that is in no way an indicator of less academic provess.

So I politely disagree.

[/ QUOTE ]

Here's what experimental physicist Vic Stenger has to say (writing in Skeptical Briefs):

-----------------------------------------------------

Higgs particles, if they exist, pervade all of space and provide an explanation for the origin of mass. In the standard model, all particles are intrinsically massless but gain mass (that is, inertia), by bouncing around off Higgs particles.

The mass of the Higgs is uncertain. But, as time has gone on, better estimates have been made. Gradually these have come down to the point where lower energy experiments currently running might have a chance to see it. Four independent experimental groups working at LEP have examined their data for signs of the lower mass Higgs. Last August, two of the groups reported that they had seen a total of four candidates at a mass of 115 GeV (the proton mass is a0.938 GeV). The other two groups had no candidates, but their data were not inconsistent.

After seeing these results, the Director-General delayed the shutdown until November 4 to give the groups time to get more data. One group found two more candidates, and a third group also reported two new ones, although these were all questionable and, in fact, one previous candidate went away.

The combined results from the four experiments were determined, by a rather complicated analysis, to have a statistical significance level of about 99.8 percent. In probability theory, this means that an effect as larger or larger than the one reported would, on average, appear as a random artifact if the same experiment were repeated 500 times.

The experimenters and much of the particle physics community pleaded with the Director-General to extend the LEP run into 2001. They argued that Fermilab, the recently upgraded U.S. accelerator located in Illinois, might discover the Higgs before LHC came on line in 2006. After "extended consultation with the appropriate scientific committees," Maiani decided to shut down LEP and proceed with LHC construction. Apparently, he was not convinced by the data presented. He promised to do his best to speed up LHC construction by asking for more resources.

It will be fascinating to see how it all plays out. In the meantime, we can use this as an excellent example of how standards vary from field to field. In fields such as medicine and psychology, a significance level of 95 percent is usually adequate for publication. Anything at 99.8 percent would be accepted as a valid observation.

I can understand the desire in health fields to get research results into therapeutic use as soon as possible. However, the 95 percent standard implies that every twentieth experiment, on the average, will report statistical artifacts as real effects. Is it any wonder that people are confused when they read media reports of studies that say one thing, only to be contradicted a few months later by studies that say the opposite?"

--------------------------------------------------

In other words, you're out to lunch, politely or no.

Entire article at: http://www.colorado.edu/philosophy/v...efs/Higgs.html

[tame_deuces]

As I stated in my post, soft sciences often has to make less bombastic statements, so I fail to see the point of your quoted article, as I was obviously well aware of the point it makes - statistical methodology in psychology is one of my study fields. Reading a post is often vital before criticising it. So I am in no way 'out to lunch'.

The important thing is that the papers are aware of this limitation, which they largely are. That popular references to them often is not (in media etc, or by amateurs on discussion boards) is a problem, but hardly blamable of the researches.

[jakeduke]

Some loosely connected thoughts on this from a grad student in biomedical engineering:

A huge part of the problem that I didn't see addressed in the OP is that negative results don't get you into Science, Nature, PNAS, etc. There is such a push for positive findings with statistical significance that results like "the treatment resulted in a small but insignificant decrease in x" aren't usually very well received, no matter how sound the explanation of the result. It also sucks real hard to work for 2+ years on a study to realize at the very end that your main hypothesis didn't pan out, so I'm sure there's a fair amount of "how can I look at my data so I can find some significance" going on.

All this being said, when I read a paper, I can (usually) tell fairly quickly if the study is well designed and if the science is good or not. So if someone's hypothesis is supported by one figure with some questionably significant data, 5 other figures of loosely related stuff and discussion sections that do lots of hedging, I'm not going to be terribly inclined to take much from the paper. So I'm not quite sure how awful this is for science in general, because the people who are reading the stuff generally have decent BS detectors in their respective fields.

[AlexM]

People, even very intelligent people, are biased, ignorant and make stupid mistakes. Of course most scientific studies are screwed up.

[AlexM]

[ QUOTE ]
[ QUOTE ]
The FDA in recent history had more drug recalls in the last few years than in all of the preceding decades put together...why?

[/ QUOTE ]

not only that, but the number of airplane crashes has gone through the roof in the last 100 years.

luckyme

[/ QUOTE ]

I theorize that global warming is caused solely by airplanes. Not by their pollution, but by the fact that the more of them there are flying around, the more it screws up the atmosphere. All that air being moved around unnaturally can't be good.

[tame_deuces]

A very good post jakeduke, and I agree with it completely and my observations are identical to yours.

[Borodog]

There has been an upsurge in criticisms of the way in which science is funded in the US and the negative consequences for science itself. There was an article recently in Nature about problems with the way NIH funds research, for example, and calling for completely reexamining how funding is awarded.

As far as my own field, the worst papers seem to make it into the popular journals. I can't recall off the top of my head if it was Science or Nature, but recently there was a paper on Mira, which was discovered to have a 4 parsec long comet-like UV tail created as Mira blasts its way through the ISM at 150 km/s or so. This paper was poor. It was an obvious example of an observational paper where the observers (who did a great job with the observations) feel pressure to take some kind of stab at a theoretical explanation of their observations. Their numbers were all wrong, they had numerous simple math mistakes and typos, they had erroneous shock jump conditions on the bowshock, and on and on. It was a 4 page paper with like 12 errors and 10 authors. 10 authors and they confuse the mass loss from Mira with the mass of a proton (I'm not making that up; it was in the paper)! It was embarrassing. We talked about it in our group meeting, since it's a system we might be very interested in simulating, but their numbers were so bad it was hard to tell which way the errors went or which side had the typos (the observations or the derived numbers). And their theory was very shakey. They conflated a couple of different velocities in a calculation. The attributed the UV tail to electrons from the hot gas from the shocked ISM exciting the cold compressed gas in the bow shock, which would then radiate in the UV. But the cooling time for the hot shocked ISM would be a few thousand years, and the flow time down the tail is a few tens of thousands of years, so that can't be it (we decided it's probably UV flourescence from the ambient UV radiation field).

Part of the problem is that the popular journals tend to have fixed reviewers. I think Nature has exactly two reviewers for all astrophysics submissions, and apparently an overbearing editor. Astrophysics is a huge field, and there is no way that two reviewers can be competent in every subfield that comes in. So basically they look for papers that are exciting, aren't the best people to be doing the peer review, and then pressure the authors to rewrite in ways that the editor wants, which is often to the detriment of the science. But being in Science and Nature is so prestigious that authors are often willing to compromise to get published there. I can't really blame them. Papers get published in Astronomy & Astrophysics or the Astrophysical Journal, or ApJ Letters, or New Astronomy, etc. all the time and you never hear about it. But make Science or Nature, and suddenly you're in the Departmental newsletter and on the front page of the University web site.

[Archon_Wing]

OP:

Indeed, blind faith in anything is bad. But that is why we must have a healthy dose of skepticism. It's good idea to actually read a study before using it as a source. [img]/images/graemlins/tongue.gif[/img]