Why do people cite surveys?
 

It is well known that the human animal is notoriously incompetent with regard to responding to surveys.

For example, in one survey, 60% of respondents claimed they were self conscious about their dandruff.

The catch? Dandruff only affects 35% of the population. Now, my precise numbers may be off, but you get the idea.

My point is that folks on this forum need to understand that most "surveys" are nonsense. This is directly in line with Borodog's point that the "what if you were..." threads are meaningless.

Re: Why do people cite surveys?
 

People often use "dandruff" as an umbrella term to describe not only dandruff but dry scalp and other similar conditions as well. I don't see how this semantic argument proves anything.

Re: Why do people cite surveys?
 

[ QUOTE ]
Why do people cite surveys?

[/ QUOTE ]

Because surveys always lie.

Re: Why do people cite surveys?
 

The fun thing is....

That you somehow take some singular pieces of anecdotal evidence about surveys as truth that they don't work is actually an extremely good argument FOR survey research.

Re: Why do people cite surveys?
 

You know what is even more funny? That you cite a survey proving your point that surveys are not reliable to prove a point.

Re: Why do people cite surveys?
 

[ QUOTE ]
You know what is even more funny? That you cite a survey proving your point that surveys are not reliable to prove a point.

[/ QUOTE ]


You know, statistically, 85% of statistics are made up on the spot.

Seriously though, and I admit this, even though I'm guilty as well. Surveys are cited because, if you really look hard enough, you can find a survey to prove anything. Take religion for example. If I were to stand outside of a church at around 12:45 on Sunday afternoon conducting a survey, I could probably get a very skewed reading on the question "Do You believe in the Genesis theory of creation?"

I interview 2000 people coming out of several different churches on several consecutive Sundays, and I have a survey that I can cite that proves that "97% of Americans believe in the Genesis theory of creation" with a sample size of 2000 interviews.

Re: Why do people cite surveys?
 

I have dandruff

Re: Why do people cite surveys?
 

[ QUOTE ]
[ QUOTE ]
You know what is even more funny? That you cite a survey proving your point that surveys are not reliable to prove a point.

[/ QUOTE ]


You know, statistically, 85% of statistics are made up on the spot.

Seriously though, and I admit this, even though I'm guilty as well. Surveys are cited because, if you really look hard enough, you can find a survey to prove anything. Take religion for example. If I were to stand outside of a church at around 12:45 on Sunday afternoon conducting a survey, I could probably get a very skewed reading on the question "Do You believe in the Genesis theory of creation?"

I interview 2000 people coming out of several different churches on several consecutive Sundays, and I have a survey that I can cite that proves that "97% of Americans believe in the Genesis theory of creation" with a sample size of 2000 interviews.

[/ QUOTE ]

This is why survey research is almost a discipline of its own, and why academic openness (as always) is vital in surveys, so you can know what means have been done not to get skewered results, and what means may give skewered results in your sample etc.

Re: Why do people cite surveys?
 

[ QUOTE ]

For example, in one survey, 60% of respondents claimed they were self conscious about their dandruff.


[/ QUOTE ]

98% of students at my middle school were very conscious of my science teacher's dandruff.

Hello Mr. Beard.

Re: Why do people cite surveys?
 

[ QUOTE ]
This is why survey research is almost a discipline of its own, and why academic openness (as always) is vital in surveys, so you can know what means have been done not to get skewered results , and what means may give skewered results in your sample etc.

[/ QUOTE ]
Photoshop material? Also, QFT. There are good surveys and bad surveys depending on the actual makeup of any representative sample, sample size, survey phrasing, order of questions, yada yada... It's important to know how a survey is conducted before it's used or accepted as any form of support.

Re: Why do people cite surveys?
 

Speaking as a practicing statistician, who has done some amount of survey design and analysis even though it's not the bulk of my work...

I blame this perception of surveys as unreliable on two things, both tied to sloppy word usage (though I am not blaming the entire problem on journalists):

1) People tend to use the same names both for uncontrolled whoever-wants-to-answer-answers or whoever-is-convenient-to-ask-gets-asked lists of questions, and for carefully administered surveys. The sampling method IS critical to the validity of the results, and all the serious pollsters and most of the casual ones are aware of this - but...

2) When surveys are in the news, the "highlights" - the most significant or surprising final results, usually - are all that gets reported. The actual published survey report is going to include literally pages of fine print; what the intended target population was, how many were selected, by what method, how nonresponse was coped with, how "I don't know" answers were coped with, and so on and on and on.

It's REALLY easy for people to accidentally (or deceptively) leap from "75% of a sample of people 18 to 45 in major east and west coast cities" to "75% of Americans." Or "75% of attendees at the American Medical Association conference" to "75% of doctors".

If you can see the report produced by the actual surveying agency, or even the press release they prepared, and you read it with a careful eye to word choice, it's usually quite easy to tell good from bad surveys apart. But the necessary information was never in the bad surveys that make it into the news, and is removed from the news reports of the good surveys that make it into the news, leaving the public (such members of the public who care) with no way to tell.

Re: Why do people cite surveys?
 

I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

Re: Why do people cite surveys?
 

[ QUOTE ]
I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

[/ QUOTE ]

If you ask men to estimate their IQ, polls say that the average answer is 137. [img]/images/graemlins/smile.gif[/img]

Re: Why do people cite surveys?
 

[ QUOTE ]
You know what is even more funny? That you cite a survey proving your point that surveys are not reliable to prove a point.

[/ QUOTE ]

Ummm, actually the point is that we can know for sure the distribution of various ailments (e.g., dandruff) in the population. So we can measure the accuracy of the response with known data. And yes, of course I made up the %'s on the spot, since I just remember reading this in a newspaper like 10 years ago.

And thanks for a good post, Siegmund. You intelligently made some points that I was just grasping at.

A question I have is how does one deal with the fact that some people just get the surveys "wrong"? For example, when people are asked about how often they do so and so (say, exercise), they might just respond incorrectly. How do you correct for this, if it is possible at all?

And the average IQ thing is lol if true...

Re: Why do people cite surveys?
 

Basically you just have to disprove of survey research which does not come from a reputable source or isn't peer referenced to similar studies and where it isn't possible to read about the sampling techniques.

Re: Why do people cite surveys?
 

[ QUOTE ]
I've been seeing this commercial on TV that claims something like, "1 in 4 people thinks they sweat more than average"

[/ QUOTE ]

There used to be one a while back that said 25% of people sweat more than average. It wasn't a joke.

Re: Why do people cite surveys?
 

It might even be true. If there are a lot of people who sweat very little, but a few people who sweat very profusely, there will indeed be only a few people who sweat more than the mean amount of moisture per day. (There will always be 50% who sweat more than the median amount per day.) A fine example of the sort of thing that gets changed when a copywriter tries to summarize a scientific study.

[ QUOTE ]

Ummm, actually the point is that we can know for sure the distribution of various ailments (e.g., dandruff) in the population.

[/ QUOTE ]

And how could we possibly know that? We'd have to estimate it, either by collecting a sample of people and seeing which ones have dandruff, or by doing something fancy with the numbers of people who see doctors for it or buy shampoos for it. Surveying, in some fashion.

It's certainly possible that two surveys on the same topic will yield different results according to how the questions are worded. There really is no way to avoid that except to test the survey on a pilot group and get some feedback on how their interpreted the questions. There are some standard rules about neutral phrasing of questions, which are a starting point.

Re: Why do people cite surveys?
 

It is obviously the case that people quite often simply do not know accurately the answer to some questions about themselves despite the fact that they are sure... but it's also the case that you can get statistically legitimate results (though the interpretation might be more difficult) from surveys. Look at the MMPI, the actual meanings to the questions don't exactly matter... but maybe that's sort of a different thing...

also lol @ the unintentionally borrowed Nietzscheism of sarcastically arguing the uselessness of supporting something with such-and-such and concluding by supporting your argument with precisely such-and-such (for as Homer says, "many lies tell the poets")