Key takeaway

FiveThirtyEight claims that early national primary polls can become more useful for predicting party nomination success if one takes candidate name recognition into account. But so few low-name recognition candidates have won their party’s nomination that you cannot reasonably predict whether a low-name recognition candidate will win their party’s nomination over a high-name recognition candidate, no matter their national polling averages. You can better predict nomination success using early national polling averages only, and you can make an even more informed prediction if you take political party into account.

Methods

I created my own datasets by copying and pasting data from tables found in FiveThirtyEight’s blog posts and at CQ Voting and Elections Collection. I’ve made those data available on my GitHub.

Early national primay polls predict nomination success and national primary vote share.

During the primary season leading up to the 2012 U.S. election, FiveThirtyEight’s Nate Silver wrote A Brief History of Primary Polling in which he claimed that early national primary polls predict who partys nominate to run in the general election. In the 3-post series (Part I, Part II, Part III), Silver analyzed national primary polling data, national primary vote share, and nomination success of both Republican and Democratic presidential candidates between 1972 and 2008. He concluded that candidates with higher (vs. lower) national primary polling averages tend to win higher national primary vote shares, and they more often win their party’s nomination. FiveThirtyEight has since updated their dataset to include primary data from the 2012 and 2016 election cycles. FiveThirtyEight’s Geoffrey Skelley used the updated dataset to reach the same conclusion in a complementary series of posts (Part I, Part II, Part III) meant to calibrate readers for the upcoming 2020 primaries.

Figure 1: The ribbon represents 95% confidence intervals around the predictions.

Figure 2: The ribbon represents 95% confidence intervals around the predictions. The vote share data are a little different from those presented by FiveThirtyEight because our sources were likely different.

Well-known candidates often win their party’s nomination.

In addition to early national primary polling, both blog series authors made use of indicators of name recognition: the proportion of primary voters who have at least heard of a given presidential candidate’s name at a given stage in the election. To estimate name recognition, they used polling questions that asked either (1) whether respodents had ever heard of a candidate or (2) whether respodents had a favorable or unfavorable opinion of a candidate. The proportion of people who had heard of a candidate or who could form an opinion (favorable or unfavorable) of a candidate served as a proxy for the proportion of people who recognize that candidate. In their most recent blog series by Skelley, FiveThirtyEight translated this proportion into a 5-point scale (20%, 40%, 60%, 80%, and 100% name recognition). They categorized those below 60% as “not well known” (low name recognition) and those above 40% as “well known” (high name recognition).

FiveThirtyEight suggests that name recognition can serve as an index of a candidate’s potential (or a kind of handicap, depending on how you look at it). The idea is that voters probably won’t report they’d vote for a candidate they don’t even recognize, so a candidate’s current polling average might underestimate what their polling average would be if more people recognized them. Of course, a candidate’s current polling average might overestimate that hypothetical value too: Voters who currently don’t recognize a candidate might not support a candidate when they eventually do recognize them.

In any case, few if any voters will actually vote for a candidate whose name they don’t recognize. So it shouldn’t be surprising that FiveThirtyEight’s data show that well known candidate’s win their party’s nomination way more often than not well known candidates. In fact, among the 19 candidates nominated by a major U.S. political party since 1972, only 4 were not well known during the early stages of the election (see Table 1 below); all of those nominees were Democrats (so none were Republicans).

FiveThirtyEight claims early national primary polling better predicts nomination success when you account for name recognition.

In their blog series, Silver and Skelley each present a table and a figure that emphasize the nomination success of candiates across different levels of early national primary polling average and name recognition. They claim that combining these pieces of information can better predict nomination success than either piece of information by itself. Given that only 4 not well known candidates have ever won their party’s nomination, you should be skeptical of some of the predicions drawn from the values displayed in these tables and figures.

In both blog series, FiveThirtyEight makes specific (numeric) predictions based on the interaction between national primary polling average and name recognition. Based on the dataset compiled before the 2012 primary elections, Silver wrote,

a candidate with 100 percent name recognition who is polling at 20 percent is roughly as likely to win his nomination as one with 50 percent name recognition who is polling at 10 percent.

And based on the the most recently compiled dataset, Skelley wrote,

And as you can see in the chart below, a low-name-recognition candidate didn’t stand much of a chance of winning unless they were able to climb past 10 percent in the polls in the first half of the year before the primaries. If they were able to hit that mark, then their odds of winning were slightly less than 1 in 4, which put them ahead of a high-name-recognition candidate polling at the same level.

These authors derived their predictions from logistic models based on the datasets available to them at the time.

Figure from Silver’s post, A Brief History of Primary Polling, Part III

Figure from Skelly’s post, We Analyzed 40 Years Of Primary Polls. Even Early On, They’re Fairly Predictive.

Regressions can make predictions where data are thin (even absent).

There is one key problem with deriving these specific predictions from logistic regressions. The problem is that the authors did not give reasons for making predictions outside the range of available data. All regressions (including logistic regressions) can make predictions based on values that are scarce or even not avaialble in the datasets used in the regression analysis. In some contexts, analysts have good reasons to believe that unobserved data that fall outside the range of observed data will follow a pattern similar to the observed data. For example, you can use regression to predict someone’s height based on their weight, even if that person is a little taller (or shorter) than anyone in the dataset you used to construct your regression equation. The same logic applies to predicting market sales based on the month of the year. But even these extrapolations are limited: Some people are much heavier than you’d expect based on their height, and sometimes sales are much lower than you’d expect based on the month of the year; such expectations can be especially misguided if they’re derived from a regression equation based on very different people or years of business.

This limitation applies equally to predicting nomination success. In the case of U.S. presidential primary nominations, there exists a limited range of national primary polling averages (e.g., no non-incumbent candidate has ever polled above 70%), and this range is even more limited among not well known candidates (i.e., no low name recognition candidate has polled above 10% in the early stage of the election). Both Silver and Skelley make predictions about the probability of nomination success for low name recognition candidates polling nationally at 10%, on average. But no low name recognition candidate polling nationally at 10% (on average) during the early stage of the election has ever won their party’s nomination. What’s more, no low name recognition Republican candidate has ever won their party’s nomination, regardless of their early national polling average. FiveThirtyEight’s figures imply even more precise predictions beyond the range of avaialble data (e.g., a low name recognition candidate polling nationally at 20%, on average, which has never happened, has an estimated 90% probability of winning). So, although these logistic regression models can make predictions that seem to make sense, these predictions use combinations of early national polling averages and name recognition values that fall outside the range of data used in the regressions themselves (see Figure 3 below). There may be good reasons to trust these specific extrapolations, but the FiveThirtyEight authors do not provide any.

Figure 3: The ribbons represent 95% confidence intervals around the predictions. The orange ribbon (not well known) envelops most of the figure because no data are available in that range.

Early national primary polls predict nomination success better without accounting for name recognition.

Given so few low name recognition candidates have won their party’s nomination (none have among Republicans), it’s difficult to justfy using name recognition to modify the already strong relationship between candidates’ early national primary polling averages and nomination success. In fact, a logistic regression that uses only candidates’ early national polling averages to predict nomination success does a slightly better job at distinguishing nomination winners from losers than a model that includes early national polling averages, name recognition, and their interaction (i.e., a variable that tests whether the predicitive power of one variable depends on another variable). In other words, FiveThityEight’s figure that emphasizes different predictions based on early national polling average and name recognition makes worse predictions than a model that relies on early national polling average alone (see Table 2 below).

Early national primary polls better predict nomination success when you account for a primary’s political party.

Candidates’ early national polling averages predict nomination success better without accounting for name recognition. However, early national polling averages predict nomination success better when you account for the major political party nominating the candidate. Put simply, differences between how Democrats and Republicans nominate candidates for the general election might affect the degree to which early national polling averages predict nomination success within either political party. For example, Republicans and Democrats employ different rules for allocating delegates in primary elections (e.g., winner take all, proportional allocation), and even within each party’s nomination process those rules depend on the state. More broadly, Republicans and Democrats often (but not always) differ on policy attitudes, moral values, personality traits, and so on; any such variables could explain the difference in the extent to which early national polls predict nomination success. Regardless of how or why early national polling averages have different predictive power, depending on the political party nominating the candidate, a model that accounts for political party simply does a better job at distinguishing nomination winners from losers than a model that does not (see Table 2 below).

Figure 4: The ribbons represent 95% confidence intervals around the predictions.

A more detailed takeaway

Much like I wrote at the start of this post, FiveThirtyEight claimed that you can better predict U.S. presidential candidate nomination success with early national primary polling averages if you also account for candidates’ name recognition. In this post, I explained that their claim requires you to extrapolate nomination success for low name recognition candidates from a model whose data includes only 4 successful low name recognition candiates. Not one of those low name recognition candidates polled above 10% (on average) in early national polls, and none were Republicans. Yet the FiveThirtyEight authors made specific (numeric) predictions about the probability that any low name recognition primary candidate (Republican or Democrat) polling at or above 10% has of winning their party’s nomination. And their figures imply predictions well beyond their 10% national polling average examples. These claims are not informed by available data, and the authors provided no reasons for why readers can rely on the extrapolations used to support their claims. I went on to explain that national polling averages can better predict nomination success without accounting for name recognition, but predictions from early national polling averages can be improved by accounting for the major political party nominating candidates.

Name recognition matters, and FiveThirtyEight makes excellent predictions.

In the spirit of clarity and collegiality, I’ll emphasize what I didn’t mean in this post. First, I didn’t mean that name recognition doesn’t matter. Name recognition does matter: High name recognition candidates are nominated by their party more often than low-name recognition candidates. Name recognition matters especially for Republicans who have never nominated a low name recognition candidate. Second, I didn’t mean that FiveThirtyEight always (or even often) extrapolates. This is one example. Third, I didn’t mean that FiveThirtyEight makes poor predictions. FiveThirtyEight makes excellent predictions (How Good Are FiveThirtyEight Forecasts); they are exceptionally transparent about thier methods (FiveThirtyEight #Methodology); and they make much of their data available for their readers to explore (FiveThirtyEight: Our Data). I’m an avid FiveThirtyEight reader and fan, so I hope anyone reading this post appreciates my criticism yet aknowledges the excellent work FiveThirtyEight does to inform (with data) its readers about politics, sports, science & health, economics, and culture.