Evaluating statistical claims
These questions don't ask you to calculate — they ask whether a study's conclusion is justified. Two features of the study design decide it: how the subjects were selected, and how they were assigned to groups.
What College Board tests
Judging whether a result can be generalized to a wider population, and whether a study can claim cause and effect. Both come down to two design choices: random selection and random assignment.
The two questions to ask
Random selection earns generalization. Random assignment earns causation. A study needs the matching design feature for the matching claim — and without it, the conclusion overreaches.
The decision in one view
| Design feature | What it licenses |
|---|---|
| Random selection from a population | Generalize the result to that population |
| Random assignment to groups | Conclude the treatment caused the effect |
| Neither (e.g. volunteers, observation) | Describe only the group studied; association at most |
Four worked examples in SAT format. Read the approach, try it yourself, then tap Show the full solution.
1 · When a result generalizes
A researcher selected 200 students at random from all students enrolled at a large university and found that 64% commute by bicycle. Which conclusion is best supported?
- About 64% of all students at the university likely commute by bicycle.
- About 64% of all college students nationwide commute by bicycle.
- Riding a bicycle causes students to enroll at the university.
- No conclusion about the university's students can be drawn.
Approach The sample was randomly selected from this university's students, so the result generalizes to that population — but no further, and it says nothing about cause.
Show the full solution
Answer: A
The 200 students were chosen at random from all students at the university, so the sample represents that population. The result can be generalized to all of the university's students — choice A.
Why the other choices are wrong
B Over-generalizes to all college students nationwide; the sample only came from this one university.
C Claims cause and effect, which no observational survey can support.
D Too cautious — random selection does justify a conclusion about the university.
2 · When a study shows cause
In a study, participants were randomly assigned to either a new study-skills program or a control group. The program group later scored higher on a test. Which conclusion is best supported?
- The program and higher scores are associated, but no cause can be claimed.
- The program likely caused the higher scores, for the participants in this study.
- All students everywhere would score higher with the program.
- Higher-scoring students chose the program.
Approach Subjects were randomly assigned to treatment vs. control, which is what licenses a cause-and-effect conclusion — for the people in the study.
Show the full solution
Answer: B
Random assignment balances the groups, so the difference in scores can be attributed to the program. The study supports a causal claim for its participants — choice B.
Why the other choices are wrong
A Understates it — random assignment does support a causal claim, not just association.
C Over-generalizes to "all students everywhere"; the result applies to the studied participants.
D Contradicts the design — participants were assigned, they didn't choose.
3 · When a result can't generalize
An online article invited readers to click a link and answer a poll about a local policy. Of those who chose to respond, 70% agreed with the policy. Why can't this be generalized to all residents?
- The sample size was too small.
- The respondents were self-selected, not a random sample of residents.
- 70% is not a high enough percentage.
- It can be generalized to all residents.
Approach Generalization requires random selection. Here, respondents opted in — a self-selected (voluntary) sample, which is not representative of all residents.
Show the full solution
Answer: B
People who choose to answer an online poll tend to differ from the general population (often those with stronger opinions). Because the sample wasn't randomly selected, the result can't be generalized — choice B.
Why the other choices are wrong
A The core problem is how the sample was formed, not its size.
C The percentage value is irrelevant to whether the sample is representative.
D A self-selected sample is exactly what blocks generalization.
4 · Association is not causation
A study observed that people who drink more coffee tend to sleep fewer hours. Participants were not assigned coffee amounts; researchers simply recorded each person's existing habits. What can be concluded?
- Drinking coffee causes people to sleep less.
- Sleeping less causes people to drink more coffee.
- There is an association between coffee intake and sleep, but no cause can be established.
- Coffee and sleep are unrelated.
Approach Nothing was randomly assigned — researchers only observed existing behavior. Observation can reveal an association, but never establish which way the cause runs (or whether a third factor drives both).
Show the full solution
Answer: C
Without random assignment, the study can only show that the two vary together. It can't tell whether coffee reduces sleep, less sleep prompts more coffee, or something else drives both — choice C.
Why the other choices are wrong
A Claims causation an observational study can't support.
B Asserts the reverse cause — equally unsupported here.
D Contradicts the data, which shows a clear association.