Need Help From Surveyors/Statisticians

[yellowjack]

I'm writing an exam for a position with Statistics Canada in the upcoming week, a lot of which is regarding problem solving with survey methodology. The resources I need are primarily available, except practical problem solving with surveying. Here is what I seek help on, directly from the website (http://www.statcan.ca/english/employ...qualifications):

<ul type="square">[*]demonstrate the ability to identify methodological problems associated with survey development and data analysis;[*]demonstrate the ability to resolve statistical and methodological problems and to apply statistical methods to practical problems associated with survey development and data analysis;[/list]
If anyone here has experience in the field could you please give some advice/knowledge on a basic approach on handling these kinds of questions? i.e. what to consider when a sample looks biased, etc. etc.? I actually don't even know what kind of questions will come up.

Thanks a lot in advance for any help you are able to provide.

[Siegmund]

I daresay you might have a little trouble with the duties of the position, if this is an area you don't know anything about.....

For a start: a) review in your scientific sampling textbook (you DID take sampling at some point, I hope? And DID keep the book?) what the common sampling methods are and the conditions under which each one is appropriate.
b) have an idea of a few of the various things that can cause bias (I won't give you a complete list - non-response bias is one of many) and
c) have an idea of what steps you can take to address bias after you suspect that it exists (there are a few biggies, and you ought to know what phrases to be googling to refresh your memory.)

[yellowjack]

Siegmund,

I appreciate your reply. I have taken one course specifically for survey sampling, but found it lacked practical applications of concepts that may be examined.

a) Ok. I'm using Sharon's Lohr's book which I have found to be very clear but have been unable to find a secondary resource. Can you recommend me another one? If you know anything about the sampling design book from the Statistics Canada catalogue please let me know.

b) Done; my list is made up of forms of selection, measurement, and nonresponse bias. Am I missing anything major?

c) This is what I'm seeking assistance of. For dealing with selection bias, the only thing I know to do is to use the biased survey for preliminary estimations in the early stages of similar investigations. This may be too simple but I am not sure what is expected. I do not know how to deal with measurement bias when it has already occurred. For nonresponse, the work-arounds are fortunately very clear.

[Siegmund]

OK, a few more comments I'll toss out...

a) I used Schaeffer, Mendenhall, and Ott when I studied sampling. Not saying there's anything special about it, I assume that most any sampling book has the formulas in it. Under "a", two questions I would want to be prepared for: discuss the conditions under which a stratified sample gives you a much better estimate than a simple random sample does (perhaps other sampling schemes too - but you'll see simple and stratified more often than any other kinds); and discuss the advantages and hazards of systematic vs. random samples.

On b, you may want to separately address measurement bias as a result of badly written questions, vs. bias resulting from respondents not answering truthfully.
You might be given a loaded question in the interview, and asked how you word reword it more fairly.
There is quite a body of literature about how to get people to feel comfortable giving honest answers to crime- or sex-related questions. Not an area I've worked in though I've read a little about it.
I have had some experience with having to ask about something easily measureable as a proxy for something that isn't - for instance, asking someone how many days of work he has missed in the past month, rather than asking how many times he has been sick in the past month.

On c, the answer to most data-related things is post-stratification or some other kind of reweighting. If there was a sampling bias, you may be able to calculate something like P(being selected under the bad sampling method)/P(being selected under the right sampling method) to use as a weight. Stick to comments on nicely established methods to avoid sounding like you are making it up on the spot for the interview, but expect to need to make it up on the spot in your real life sometime.