Why do people cite surveys?

[Siegmund]

Speaking as a practicing statistician, who has done some amount of survey design and analysis even though it's not the bulk of my work...

I blame this perception of surveys as unreliable on two things, both tied to sloppy word usage (though I am not blaming the entire problem on journalists):

1) People tend to use the same names both for uncontrolled whoever-wants-to-answer-answers or whoever-is-convenient-to-ask-gets-asked lists of questions, and for carefully administered surveys. The sampling method IS critical to the validity of the results, and all the serious pollsters and most of the casual ones are aware of this - but...

2) When surveys are in the news, the "highlights" - the most significant or surprising final results, usually - are all that gets reported. The actual published survey report is going to include literally pages of fine print; what the intended target population was, how many were selected, by what method, how nonresponse was coped with, how "I don't know" answers were coped with, and so on and on and on.

It's REALLY easy for people to accidentally (or deceptively) leap from "75% of a sample of people 18 to 45 in major east and west coast cities" to "75% of Americans." Or "75% of attendees at the American Medical Association conference" to "75% of doctors".

If you can see the report produced by the actual surveying agency, or even the press release they prepared, and you read it with a careful eye to word choice, it's usually quite easy to tell good from bad surveys apart. But the necessary information was never in the bad surveys that make it into the news, and is removed from the news reports of the good surveys that make it into the news, leaving the public (such members of the public who care) with no way to tell.

[Johnny Drama]

I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

[Jamougha]

[ QUOTE ]
I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

[/ QUOTE ]

If you ask men to estimate their IQ, polls say that the average answer is 137. [img]/images/graemlins/smile.gif[/img]

[borisp]

[ QUOTE ]
You know what is even more funny? That you cite a survey proving your point that surveys are not reliable to prove a point.

[/ QUOTE ]

Ummm, actually the point is that we can know for sure the distribution of various ailments (e.g., dandruff) in the population. So we can measure the accuracy of the response with known data. And yes, of course I made up the %'s on the spot, since I just remember reading this in a newspaper like 10 years ago.

And thanks for a good post, Siegmund. You intelligently made some points that I was just grasping at.

A question I have is how does one deal with the fact that some people just get the surveys "wrong"? For example, when people are asked about how often they do so and so (say, exercise), they might just respond incorrectly. How do you correct for this, if it is possible at all?

And the average IQ thing is lol if true...

[tame_deuces]

Basically you just have to disprove of survey research which does not come from a reputable source or isn't peer referenced to similar studies and where it isn't possible to read about the sampling techniques.

[xSCWx]

[ QUOTE ]
I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

[/ QUOTE ]

There used to be one a while back that said 25% of people sweat more than average. It wasn't a joke.

[Siegmund]

It might even be true. If there are a lot of people who sweat very little, but a few people who sweat very profusely, there will indeed be only a few people who sweat more than the mean amount of moisture per day. (There will always be 50% who sweat more than the median amount per day.) A fine example of the sort of thing that gets changed when a copywriter tries to summarize a scientific study.

[ QUOTE ]

Ummm, actually the point is that we can know for sure the distribution of various ailments (e.g., dandruff) in the population.

[/ QUOTE ]

And how could we possibly know that? We'd have to estimate it, either by collecting a sample of people and seeing which ones have dandruff, or by doing something fancy with the numbers of people who see doctors for it or buy shampoos for it. Surveying, in some fashion.

It's certainly possible that two surveys on the same topic will yield different results according to how the questions are worded. There really is no way to avoid that except to test the survey on a pilot group and get some feedback on how their interpreted the questions. There are some standard rules about neutral phrasing of questions, which are a starting point.

[mbillie1]

It is obviously the case that people quite often simply do not know accurately the answer to some questions about themselves despite the fact that they are sure... but it's also the case that you can get statistically legitimate results (though the interpretation might be more difficult) from surveys. Look at the MMPI, the actual meanings to the questions don't exactly matter... but maybe that's sort of a different thing...

also lol @ the unintentionally borrowed Nietzscheism of sarcastically arguing the uselessness of supporting something with such-and-such and concluding by supporting your argument with precisely such-and-such (for as Homer says, "many lies tell the poets")