A political scientist wants to predict how the residents of New Jersey will react to a new bill. Which study design will provide the most reliable results?
A) Mailing a questionnaire to each of 200 randomly selected residents of NJ
B) Surveying a group of 300 randomly selected NJ residents
C) Interviewing a group of randomly selected students from a NJ public university
D) Surveying 1500 randomly selected US residents
The answer is B, but why wouldn’t it be A?
Let me first answer your question with a question: why wouldn’t it be B? It has all the elements of A that are good (random selection, sufficiently large sample, targeted to the population the researcher cares about) but the sample in B is larger. Why wouldn’t you choose B over A?
I think the real problem with A is that you don’t know how many questionnaires you’re going to get back when you mail 200 out (I wouldn’t be surprised if the return rate on something like that is way low), but I don’t really think a real SAT math question would ask you to think through real-world logistics to that extent.
Luckily, you need not rely only on your intuition about questionnaire return rates—you can also rely on your knowledge that a larger sample size will always provide marginally more reliable results, so even if you don’t see a difference between a mailed questionnaire and a survey, you can choose based on the difference between 200 and 300.