Convenience Samples
POLSCI 4SS3
Winter 2024
Course so far
Representative surveys as the gold standard
Other research design help us learn more but tend to use non-representative samples
Today: Talk about convenience samples
We are seeing less of this
And more of this
Or this
Do we want surveys to be representative?
Pros?
Cons?
We always want them!
But when do we need them?
Rather, when can we get away with not having them?
Internal an external validity
Validity: Approximate truth or usefulness of an inference
Inference: How we interpret the results of a study
Internal validity: Whether inferences from a single study cannot be explained by other factors
External validity: Whether inferences from a single study apply to a broader population or other target populations
Convenience samples make it easier to achieve internal validity at the expense of external validity
Types of internal validity
X-validity
(endogenous variables)
T-validity
(treatments, conditions)
Y-validity
(outcome variables)
C-validity
(context)
See Egami and Hartman (2023) for more
X-validity
Is the sample comparable to the target population?
If not, can we claim that the differences can be ignored?
To do that, we have to convince ourselves that:
. . .
- Effects are the same across units
. . .
OR
. . .
- We observe all the variables that may explain discrepancies in effects
T-validity
Do treatments
(conditions)
reflect what participants would encounter in the real world?Example: Is thinking about hypothetical countries a good reflection to how people would think about real countries?
Can we claim that there are no different versions of the same treatment?
To do that, we need to convince ourselves that everyone would interpret vignettes in the same way
Either because it is realistic enough or abstract yet believable
Y-validity
Do the outcomes we measure in surveys reflect the outcomes we want to learn about in the real world?
Example: Are self-reported vote intentions a good replacement for actual voting behavior?
Can we claim that there are no different versions of the same outcome?
Need to convince ourselves that measured outcomes are sufficiently valid and reliable
C-validity
Do results generalize from other contexts?
Example: If it worked with students in Sweden, will it work with students in Canada?
Can we claim that the same units would react in the same way if the study was conducted elsewhere?
Need to convince ourselves that context is irrelevant for similar people in different places
Discussion
Munger et al (2021): Accessibility and generalizability
- Replicate 3 convenience sample survey experiments with representative sample
Social commentary and news source credibility
Facebook shares and news consumption
Issue framing and support for gun control
- Argument: Effects vary considerably by age and digital literacy
Findings
Replication 1: Participants low on digital literacy did not respond differently to vignettes
Replication 2: Older people clicked on whatever headline came first
Replication 3: No differences because issue had nothing to do with digital literacy
What kind of validity is this about?
Coppock et al (2018): Generalizability of heterogeneous treatment effect estimates across samples
Replicate 27 studies from nationally-representative samples with convenience samples
Compare how effects vary across 16 demographic characteristics
Explanation
- Different samples yield similar results when:
Treatment effects are mostly homogeneous
Effect heterogeneity is orthogonal to sample selection
- What type of validity is this about?
After Recess
Evidence-Informed Policy
Focus on: New topic!
Break time!