POLSCI 4SS3
Winter 2024
Policy is an umbrella term to describe government programs or operations at different levels
Examples:
How long should form 57B be?
Should we get help from private clinics to clear surgey backlogs?
Should the education budget increase?
When should the next federal election be held?
Of course we want to base policy on evidence!
But there is no objective evidence when it comes to human behavior
We say evidenceinformed because the best we can do is try to prove ourselves wrong, but we cannot base policy on evidence the same way medicine does
A lady declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup
How do you evaluate this claim?
Suppose we have eight milk tea cups
4 milk first, 4 tea first
We arrange them in random order
Lady knows there are 4 of each, but not which ones
True Order



Lady's Guesses  Tea First  Milk First 
Tea First  3  1 
Milk First  1  3 
She gets it right \(6/8\) times
What can we conclude?
How does “being able to discriminate” look like?
Same for policy, we don’t know how the world where the policy works look like
But we do know how a person without the ability to discriminate milk/tea order looks like
This lets us make probability statements about this hypothetical world of no effect
Count  Possible combinations  Total 

0  xxxx  \(1 \times 1 = 1\) 
1  xxxo, xxox, xoxx, oxxx  \(4 \times 4 = 16\) 
2  xxoo, xoxo, xoox, oxox, ooxx, oxxo  \(6 \times 6 = 36\) 
3  xooo, oxoo, ooxo, ooox  \(4 \times 4 = 16\) 
4  oooo  \(1 \times 1 = 1\) 
Count  Possible combinations  Total 

0  xxxx  \(1 \times 1 = 1\) 
1  xxxo, xxox, xoxx, oxxx  \(4 \times 4 = 16\) 
2  xxoo, xoxo, xoox, oxox, ooxx, oxxo  \(6 \times 6 = 36\) 
3  xooo, oxoo, ooxo, ooox  \(4 \times 4 = 16\) 
4  oooo  \(1 \times 1 = 1\) 
Count  Possible combinations  Total 

0  xxxx  \(1 \times 1 = 1\) 
1  xxxo, xxox, xoxx, oxxx  \(4 \times 4 = 16\) 
2  xxoo, xoxo, xoox, oxox, ooxx, oxxo  \(6 \times 6 = 36\) 
3  xooo, oxoo, ooxo, ooox  \(4 \times 4 = 16\) 
4  oooo  \(1 \times 1 = 1\) 
Count  Possible combinations  Total 

0  xxxx  \(1 \times 1 = 1\) 
1  xxxo, xxox, xoxx, oxxx  \(4 \times 4 = 16\) 
2  xxoo, xoxo, xoox, oxox, ooxx, oxxo  \(6 \times 6 = 36\) 
3  xooo, oxoo, ooxo, ooox  \(4 \times 4 = 16\) 
4  oooo  \(1 \times 1 = 1\) 
If the lady is not able to discriminate milktea order, the chance of observing 6/8 correct guesses or better is 24%
We can translate this to general statements about policies or experiments
If the null hypothesis of no effect is true…
… the pvalue is the probability of observing a result equal or more extreme than what is originally observed
Smaller pvalues give more evidence against the null, which helps us make a case for the policy having an effect
A convention in the social sciences is to claim that something with \(p < 0.05\) is statistically significant^{1}
Committing to a significance level implies accepting that sometimes we will get \(p < 0.05\) by chance
This is a false positive result
A good answer strategy as a controlled false positive rate (more in the lab!)
Focus on: Research design alternatives