Sensitive Questions
POLSCI 4SS3
Winter 2024
Last week
Representative surveys as the gold standard of public opinion research
Challenging to decide:
- What to ask
- Who to ask
Today
A little more about data strategies
… and how they translate into answer strategies
Research designs for sensitive questions
Lab: Practice writing research designs on your own
Data Strategies
Elements of a data strategy
Sampling
Treatment assignment
Measurement
Elements of a data strategy
Sampling
(last week)
Treatment assignment
(next week)
Measurement
(TODAY)
Measurement as multiple independent draws
Sensitive Questions
Why do people lie in surveys?
Inattention
Satisficing
Limited options
Demand effects
Sensitivity bias
Why do people lie in surveys?
Inattention
Satisficing
Limited options
Demand effects
Sensitivity bias
How to prevent lying?
Add noise to the question
Two approaches:
Distract from the sensitive attitude/behavior
Guarantee anonymity
- Different designs vary on how they combine the two
Randomized response
For this question, I want you to answer yes or no.
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw.
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no.
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes.
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
. . .
[TURN AWAY FROM RESPONDENT]
. . .
Now you throw the dice so that I cannot see what comes out.
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
[TURN AWAY FROM RESPONDENT]
Have you thrown the dice?
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
[TURN AWAY FROM RESPONDENT]
Have you picked it up?
Randomized response
For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.
Now, during the height of the conflict in 2007 and 2008 (in Afghanistan), did you know any militants, like a family member, a friend, or someone you talked to on a regular basis?
. . .
Please, before you answer, take note of the number you rolled on the dice.
Answer strategy
We know that about \(1/6 \approx 0.17\) respondents said yes because they rolled a
So if \(30\%\) in total said yes
We have \(\widehat{Y} = 0.3 - 0.17 = 0.13\) as our population estimate
But we do not know who they are in our survey!
Assumptions
. . .
People respond honestly when guaranteed anonymity.
. . .
Those who do not hold the sensitive trait never falsely claim to bear it.
. . .
- These cannot be verified with data!
List experiment
Now I am going to read you three things that make people angry or upset.
List experiment
Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you.
List experiment
Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.
. . .
Control group
- The federal government increasing the tax on gasoline
- Professional athletes getting million-dollar contracts
- Large corporations polluting the environment
List experiment
Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.
Treatment group
- The federal government increasing the tax on gasoline
- Professional athletes getting million-dollar contracts
- Large corporations polluting the environment
- A black family moving next door
Answer strategy
Respondents are randomly assigned to conditions
Differences in responses can only be attributed to the presence/absence of the sensitive item
So \(\widehat{Y} = \text{Mean(treatment)} - \text{Mean(control)}\) our prevalence rate estimator
But we do not know who they are in our survey!
Assumptions
. . .
Those who do not hold the sensitive item never falsely claim to bear it.
. . .
Including the sensitive item does not change how participants respond to the baseline items
. . .
- These can only be evaluated indirectly
Other research designs
Building trust with respondents
Variants of the randomized response and list experiment
Example
Vignette
Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.
Vignette
- The federal government increasing assistance to the poor
- Professional athletes making millions of dollars per year
- Large corporations polluting the environment
. . .
Treatment 1
4. Granting citizenship to a legal immigrant who is Muslim
Treatment 2
4. Granting citizenship to a legal immigrant who is Christian
Next Week
Survey Experiments
Focus on: What are they good for?
Break time!