Why you shouldn’t read too much into small unweighted subsamples

3/10/2024 TSE Comments 0 Comment

Here are our two polls of KY-06, the district where we closed out live polling at KY-06.
Look at the difference, by chance, at n=100! These are two *weighted* samples with a net-30 point difference. In the end, totally the same. pic.twitter.com/P4m7Lh8sko
— Nate Cohn (@Nate_Cohn) September 29, 2024

In crosstabs, the subgroups aren't weighted. They don't even have the same number of Dems/Reps from poll to poll, even though the overall number across the full sample is the same. The weighting necessary to balance a sample overall can sometimes even distort a subgroup further
— Nate Cohn (@Nate_Cohn) September 29, 2024

That was very much behind the decision to do live polling back in the day. We were going to show you how the sausage gets made, you were going to see that it was imperfect and gross, and yet it miraculously it was still going to be reasonably useful
— Nate Cohn (@Nate_Cohn) September 29, 2024

It's worth noting that this ~n=700 range for state polls corresponds to a fairly optimal point for trying to balance individual states v. coverage the whole battleground, even though doubling would be huge for xtabs (and why our national polls are way bigger) pic.twitter.com/XmpFGqKSZH
— Nate Cohn (@Nate_Cohn) September 29, 2024

This is a very usual thread about why you should never read too much in to unweighted subsamples, this is a universality that applies to polls across the world, from Kentucky to Kirkintilloch and everywhere else inbetween.

TSE

politicalbetting.com

The web's premier resource for political betting.

Why you shouldn’t read too much into small unweighted subsamples

3/10/2024 TSE Comments 0 Comment