Sensitive Questions

 

POLSCI 4SS3
Winter 2024

Last week

  • Representative surveys as the gold standard of public opinion research

  • Challenging to decide:

    • What to ask
    • Who to ask

Today

  • A little more about data strategies

  • … and how they translate into answer strategies

  • Research designs for sensitive questions

  • Lab: Practice writing research designs on your own

Data Strategies

Elements of a data strategy

  1. Sampling

  2. Treatment assignment

  3. Measurement

Elements of a data strategy

  1. Sampling (last week)

  2. Treatment assignment (next week)

  3. Measurement (TODAY)

Measurement as multiple independent draws

Sensitive Questions

Why do people lie in surveys?

  1. Inattention

  2. Satisficing

  3. Limited options

  4. Demand effects

  5. Sensitivity bias

Why do people lie in surveys?

  1. Inattention

  2. Satisficing

  3. Limited options

  4. Demand effects

  5. Sensitivity bias

How to prevent lying?

  • Add noise to the question

  • Two approaches:

  1. Distract from the sensitive attitude/behavior

  2. Guarantee anonymity

  • Different designs vary on how they combine the two

Randomized response

For this question, I want you to answer yes or no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

. . .

 

[TURN AWAY FROM RESPONDENT]

. . .

 

Now you throw the dice so that I cannot see what comes out.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you thrown the dice?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you picked it up?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

Now, during the height of the conflict in 2007 and 2008 (in Afghanistan), did you know any militants, like a family member, a friend, or someone you talked to on a regular basis?

. . .

Please, before you answer, take note of the number you rolled on the dice.

Answer strategy

  • We know that about \(1/6 \approx 0.17\) respondents said yes because they rolled a

  • So if \(30\%\) in total said yes

  • We have \(\widehat{Y} = 0.3 - 0.17 = 0.13\) as our population estimate

  • But we do not know who they are in our survey!

Assumptions

. . .

1. Honesty given protection

People respond honestly when guaranteed anonymity.

. . .

2. One-sided lying

Those who do not hold the sensitive trait never falsely claim to bear it.

. . .

  • These cannot be verified with data!

List experiment

Now I am going to read you three things that make people angry or upset.

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you.

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.

. . .

Control group

  1. The federal government increasing the tax on gasoline
  2. Professional athletes getting million-dollar contracts
  3. Large corporations polluting the environment

List experiment

Now I am going to read you three things that make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY.

Treatment group

  1. The federal government increasing the tax on gasoline
  2. Professional athletes getting million-dollar contracts
  3. Large corporations polluting the environment
  4. A black family moving next door

Answer strategy

  • Respondents are randomly assigned to conditions

  • Differences in responses can only be attributed to the presence/absence of the sensitive item

  • So \(\widehat{Y} = \text{Mean(treatment)} - \text{Mean(control)}\) our prevalence rate estimator

  • But we do not know who they are in our survey!

Assumptions

. . .

1. No liars

Those who do not hold the sensitive item never falsely claim to bear it.

. . .

2. No design effects

Including the sensitive item does not change how participants respond to the baseline items

. . .

  • These can only be evaluated indirectly

Other research designs

Example

Vignette

Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.

Vignette

  1. The federal government increasing assistance to the poor
  2. Professional athletes making millions of dollars per year
  3. Large corporations polluting the environment

. . .

Treatment 1

4. Granting citizenship to a legal immigrant who is Muslim

Treatment 2

4. Granting citizenship to a legal immigrant who is Christian

Next Week

Survey Experiments

Focus on: What are they good for?

Break time!

 

Lab