Estimating Vote Choice: A New Look at the Generic Congressional Ballot

Voters typically pay little attention to the campaign for the House of Representatives in their district. This has complicated efforts to tap public opinion about these races because seemingly modest changes in the order or wording of questions can produce dramatically different responses (Box-Steffensmeier, Jacobson, and Grant 2000; Gow and Eubank 1984; Mann 1978; Mann and Wolfinger 1980; Wright 1993).¹ Incumbency causes particular problems. It has been shown that placing too much emphasis on candidate names can bias estimates of vote choice in favor of incumbents because these incumbents are so much better known than their challengers. Voters recognize the incumbent’s name and seize on it, ignoring the partisan considerations that play a larger role in the voting booth (Box-Steffensmeier, Jacobson, and Grant 2000).

This evidence is compelling, but it applies to only a fraction of congressional election surveys. Most surveys can not include candidate names at all because adding the names requires identifying a respondent’s congressional district before the interview. Instead, surveys that sample more than one district typically use a variant of the “generic congressional ballot.” As the name would suggest, the generic ballot ignores the specifics of each race by referring only to party labels. Gallup asks the question this way: “If the elections for Congress were being held today, which party’s candidate would you vote for in your congressional district—the Democratic Party’s candidate or the Republican Party’s candidate?” (Gallup Poll 2000)². The question is frequently used to estimate national party strength and to predict the national vote swing in the coming congressional elections. Yet little if anything is known about the accuracy of the question for the districts it samples. Does eliminating candidate names produce an estimate that favors challengers, just as questions that emphasize candidate names favor incumbents? Examining the accuracy of the generic ballot would provide a rare look into the utility and limits of a workhorse of public opinion research. It would also shed light on the other instrument effects identified by the congressional elections literature and on the dynamics of House races more generally.

Using a study from the 2000 election in California, we demonstrate that the generic congressional ballot seriously understates support for incumbents. We offer some evidence that this underestimate stems from the voting patterns of challenger partisans. These partisans are more likely to support the incumbent when prompted with his or her name and are more likely to support their own party when only party labels are provided as a cue. We also examine this question-wording effect separately by the competitiveness of the local race and discover that the bias is smaller in races with high-spending challengers. This finding is consistent with arguments that voting on election day is more partisan in these competitive races. In short, the absence of candidate names saps House races of the personal dynamic that helps make incumbents imposing political opponents. In so doing, it overstates the case for the challenger.

Bias in the National Election Study

The bias in the congressional battery of the National Election Study (NES) is a good place to develop expectations about the accuracy of the generic congressional ballot. Prior to 1978, the NES asked respondents: “How about the vote for Congressman—that is, for the House of Representatives in Washington? Did you vote for a candidate for Congress? [IF YES: Who did you vote for? Which party was that?]” (Box-Steffensmeier, Jacobson, and Grant 2000, p. 260). This wording produced reasonably accurate estimates of election returns in the districts surveyed, but many students of public opinion—based in part on the research of Mann (1978)—began to worry that the question relied too much on the ability of respondents to recall candidate names. To address this potential problem, the format was changed as part of a more detailed battery of congressional elections questions prepared for the 1978 study. In place of the original question, respondents were shown a “sample ballot” with the names and party affiliations of the candidates. They were then asked: “Here is a list of candidates for major races in this district. How about the election for the House of Representatives in Washington. Did you vote for a candidate for the U.S. House of Representatives? [IF YES: Who did you vote for?]” (Box Steffensmeier, Jacobson, and Grant 2000, pp. 260–61).

Instead of producing more accurate estimates, this wording seriously inflated support for the incumbent. Box-Steffensmeier, Jacobson, and Grant (2000) provide convincing evidence that this pro-incumbent bias stemmed from the de-emphasis on party and the overemphasis on candidate names in the sample ballot. When party labels are salient—as they are in the voting booth—the respondent gives more weight to partisan considerations. But on the NES sample ballot the names received top billing, which proved enough to lead many respondents to report voting for the incumbent. This pro-incumbent bias is larger for challenger partisans, who are more likely to support their own party’s candidate when party labels are prominent but will turn to the better-known incumbent when partisan considerations are no longer salient.

Candidate names and party labels produce distinct effects because each emphasizes a different aspect of the race. Candidate names frame the election as a contest between two individuals. This makes it no contest at all: the challenger is usually unknown and unskilled, and the incumbent is well known and well liked (Mann and Wolfinger 1980). Party labels, by contrast, present the election as the local manifestation of a clash between national political forces. Though the challenger may still lose on these grounds, he or she is in a relatively better position; be now, as part of a larger political movement, the challenger can at least rely on loyal party voters in the district for support.

On election day, both the personal and the partisan come into play. Indeed, Box-Steffensmeier, Jacobson, and Grant (2000) suggest the NES estimates were more accurate prior to 1978 because they allowed respondents to vote on whatever basis they chose: by name, if they could remember a candidate’s name, or by party, if the name did not come to them. They showed that respondents naming a valid candidate in the “Who did you vote for?” portion of the question voted disproportionately for the incumbent, while those relying on the “Which party was that?” portion tilted strongly to the challenger. The two effects balanced out, producing an accurate estimate of the overall vote.

Thus, if the NES is any indication, the generic congressional ballot—which removes names entirely—should inflate support for challengers. Challenger partisans will be encouraged to support their own party, even though in the voting booth the incumbent’s more familiar name will lure many of them to support him or her instead. The result should be an underestimate of the vote for incumbents, even as the NES question overestimated that same support. We turn now to data that can be used to test this prediction.

Data

Ideally, we would like to use a split sample design: respondents could be randomly assigned to receive either the generic congressional ballot or a question with candidate names. Unfortunately, at this writing we know of no such survey experiment. Instead, we use data collected by the Public Policy Institute of California (PPIC) during the 2000 election cycle and compare the self-reports to actual election returns. Though this makes our findings necessarily preliminary, the results strongly match expectations from other research and so make us confident in our interpretation of the results.

The data set is composed of two large-scale surveys of the California electorate—one in September and one in October—that have been merged together (see Baldassare 2000a, 2000b). Because these surveys were designed for a number of purposes other than congressional research, the sample was not matched to congressional districts prior to interviewing. As a result, the congressional vote was measured with a variant of the generic congressional ballot: “If the election for the U.S. House of Representatives were being held today, would you vote for the Republican Party’s candidate or the Democratic Party’s candidate for the House in your district?”

The PPIC survey also asked for each respondent’s zip code, which allowed us to merge congressional district–level data, including the party of the incumbent, into the data set with the telephone interview sample.³ A combination of missing data on the zip code question and overlapping boundaries among zip codes and congressional districts meant that we could accurately identify the district for 77 percent of the interview cases.⁴ Working with this subset did not appear to introduce any obvious biases (see table A1 in the appendix for a comparison of demographic statistics).

Table A1.

PPIC Statewide Survey Demographics, With and Without Unidentified Cases, Registered Voters Only

	Sample (%)
	Identified Only*	All Cases
Age
18 to 24	10.01	10.11
25 to 34	17.99	17.63
35 to 44	21.27	21.41
45 to 54	20.74	20.47
55 to 64	13.00	12.97
65 or older	16.59	16.54
Refuse to answer	0.40	0.88
Gender
Male	48.01	49.00
Female	51.99	51.00
Region
Central Valley	21.11	19.89
San Francisco Bay Area	19.10	18.67
Los Angeles	22.46	24.26
Other Southern California	25.47	26.63
Other	11.86	10.54
Education
Some high school or less	5.63	6.03
High school graduate	21.03	20.73
Some college	31.10	31.05
College graduate	26.66	26.47
Postgraduate degree	15.40	15.07
Refuse to answer	0.18	0.65
Income
Under $20,000	11.52	11.74
$20,000 to $39,999	22.51	22.10
$40,000 to $59,999	19.50	19.12
$60,000 to $79,999	13.87	13.58
$80,000 to $99,999	8.06	7.90
$100,000 or more	14.24	13.93
Refuse to answer	10.30	11.62
Race/Ethnicity
Asian American	4.33	4.49
Black/African American	5.28	5.54
Hispanic/Latino	15.69	16.93
Non-Hispanic white	71.07	68.55
Other	1.27	1.37
Party
Democrat	44.12	45.02
Republican	37.78	36.83
Other	4.31	4.33
Independent	13.79	13.83
N	(3,758)	(4,896)

^*

Refers to identification of a respondent’s congressional district.

We further narrowed the data set with three criteria. First, we eliminated uncontested seats, where respondents were not presented with a choice. Second, we eliminated the one open seat contested in the 2000 election in California. Finally, we dropped unregistered respondents to better approximate the population of voters. After excluding all of these cases, we were left with 1,941 for most purposes. This gives us district-level variance that is useful in the analyses reported below.

Analysis

How well did the PPIC survey predict the actual split in the two-party vote? Table 1 starts with the estimated Democratic vote share from intentions reported in the telephone interviews, compared to the percentage that Democrats actually received on election day in the districts sampled. The critical column in this table is the last one, which displays the difference between the estimates and the actual returns. The PPIC survey underreported the Democratic vote by about 2 percentage points, a difference that is modest and not statistically significant. This finding suggests that the survey slightly overrepresented Republican strength but otherwise performed reasonably well as a gauge of the House vote in California.

Table 1.

Actual and Estimated House Vote and Partisan Registration

	Estimated	Actual	Difference+
Voted for the Democrat (%) N = 1,941	54.3	56.3	−2.0#
Voted for the Incumbent (%) N = 1,941	58.8	67.6	−8.8***
Registered with incumbent’s party (%) N = 2,243	47.5	49.2	−1.7

Source.—Public Policy Institute of California Statewide Survey.

Note.—Respondents who chose third-party candidates have been excluded from the calculation of vote choice, but respondents who identified with third parties or as independents have not been excluded from the calculation of registration. Because the numbers are meant to estimate the actual registration figures, leaning independents have been left as independents for the party registration calculation. Analysis includes only districts with contested incumbents.

⁺

Estimated percentage minus actual percentage.

^#

p < .10 (two-tailed test).

^***

p < .001 (two-tailed test).

However, our argument is that the estimates will have a bias against incumbents, not against either party in particular. Thus, a better way to look at the numbers is to examine support for the incumbent, whether Democratic or Republican, and compare this support to incumbent performance in the election returns. This comparison also appears in table 1, and it makes the PPIC estimates appear much worse: they are 8.8 percentage points off the actual election returns, compared to just 2.0 points for the party estimates. Moreover, the bias is negative, meaning the results systematically underestimate incumbent support. Nor is this PPIC bias the result of a flawed sample. As the last row of table 1 shows, the PPIC estimates of incumbent party registration are only 1.7 percentage points lower than the numbers reported by the California Secretary of State, a difference that is not statistically significant.⁵

These findings are exactly what we would predict if the bias in vote estimation were a question-wording effect related to incumbency. The wording in the PPIC questionnaire emphasizes party and so encourages a partisan response, while incumbency, in reality, plays a significant role in most congressional campaigns. Voters frequently cross party lines to support the incumbent, who has large advantages in money, skill, and name recognition. The generic ballot—which emphasizes partisanship above all other cues—misses this cross-party vote.

Yet not all House elections are uncompetitive. Though most campaigns feature a strong incumbent and a weak challenger, a few are more evenly matched. Furthermore, it is well established in the literature that this competitive variance can be measured fairly well with either challenger political experience, challenger spending, or both (Green and Krasno 1988; Jacobson 1978, 1989, 1990; Jacobson and Kernell 1990; Johannes and McAdams 1981; Krasno 1994; Westlye 1991). We can use this district-level variance in competition to our analytical advantage. If challenger partisans turn to the incumbent in uncompetitive races when information about their own candidate is scarce, it follows that in competitive races—where information about the challenger is more readily available—defection among challenger partisans to the incumbent should be lower (Jacobson 1990; Jacobson and Kernell 1990). By the same logic, if the PPIC estimates are biased against incumbents because they fail to reflect accurately the support usually afforded incumbents by challenger partisans, the estimates should be more accurate in competitive races where the support from challenger partisans is lower anyway.

Competitive districts are rare, so it is sometimes difficult to find enough cases for analysis. The PPIC survey avoids this problem by accident because the 52 California congressional races in 2000 included several of the most competitive in the country. We define competitive races by relative spending: the proportion of the total two-party spending in each race that is accounted for by the challenger. We call a race “hard-fought” if the challenger spent more than 40 percent of the total, and “low-key” otherwise (with a nod to Westlye 1991). Table 2 shows why 40 percent is a sensible standard: based on congressional campaigns since 1982, it seems roughly the amount a challenger needs in order to wage a competitive campaign. Challengers accounting for more than 40 percent of the spending have averaged about 43 percent of the vote, with 45 percent or better a common standard for a competitive race. The 40 percent spending cutoff also marks an increase in the percentage of challengers who win, from roughly 10 percent to 22 percent.⁶ Applying this standard to California House races led us to classify eight districts as “hard-fought” and the rest as “low-key.” We tried other standards for competitiveness, and they did not change the results.⁷

Table 2.

Spending and Election Outcome, 1982–2000

	Challenger’s Share of Spending
	<10%	10%–;20%	20%–;30%	30%–;40%	40%–;50%	>50%
Challenger’s Share of Vote (%)	26.4	33.8	37.7	41.3	43.5	43.3
Challenger Victories (%)	0.1	1.0	2.7	9.6	21.8	24.0
N	(1,339)	(482)	(402)	(394)	(257)	(150)

Sources.—Gary Jacobson, University of California, San Diego; Federal Election Commission.

Table 3 displays the comparison of estimates from the PPIC survey to actual returns, split by our measure of district competitiveness. As before, the critical columns are the ones with the difference between the actual returns and the estimates. Among uncompetitive districts, the results are largely the same: the incumbent vote falls short of the actual outcome by 9 percentage points, compared to 8.8 points for all districts. But among districts with hot races, we see a confirmation of our theoretical prediction. The estimates are remarkably accurate—deviating by a statistically insignificant 0.9 points in favor of incumbents.⁸ Sampling bias is not a problem here either: the second row of table 3 separates party registration figures by competitiveness and shows that the PPIC estimates are just as accurate in the hard-fought races as in the low-key ones. Incumbent partisan registration is only 1.6 percentage points low in races where the challenger was weak, and only 2.4 percentage points high in races where the challenger was strong.

Table 3.

Actual and Estimated House Vote and Partisan Registration, by District Competitiveness

	Low-Key Races					Hard-Fought Races
	Estimated	Actual	Difference+	Estimated	Actual	Difference+
Voted for Incumbent PPIC (%)	61.4	70.4	−9.0***	49.2	50.1	−0.9
N	(1,527)			(414)
Registered with incumbent’s party (PPIC) (%)	49.2	50.8	−1.6*	41.0	38.6	2.4
N	(1,779)			(464)

Sources.—Public Policy Institute of California Statewide Survey, California Secretary of State.

Note.—Respondents who chose third-party candidates have been excluded from the calculation of vote choice, but respondents who identified with third parties or as independents have not been excluded from the calculation of registration. Because the numbers are meant to estimate the actual registration figures, leaning independents have been left as independents for the party registration calculation. Analysis includes only registered voters in districts with contested incumbents.

⁺

Estimated minus actual.

^*

p < .05 (one sample, two-tailed t-test)

^***

p < .001 (one sample, two-tailed t-test).

We can take our analysis one step further. If our expectations are correct, on election day challenger partisans often defect to the incumbent in low-key races and stick by their party in hard-fought ones. The PPIC survey deprives respondents of the cue they need to follow this pattern: without candidate names they may have trouble identifying the incumbent, or they may be encouraged to think in partisan terms. Either way, the PPIC survey should predict the same partisan support in hard-fought and low-key races. As a test of this idea, table 4 shows the percentage of challenger and incumbent identifiers who defect—that is, the percentage in each party who vote against their partisan identification.⁹ Our predictions are largely confirmed. The PPIC survey suggests that partisans defect at a similarly low rate in all races. Support for the incumbent is very high among incumbent partisans: only 7 percent choose the challenger regardless of the competitiveness of the race. Challenger partisans likewise show strong support for the challenger, with only 12 percent voting for the incumbent in low-key races and 10 percent voting for him or her in hard-fought races.

Table 4.

Defection by Party, District Competitiveness, and Survey

	Low-Key	Hard-Fought
Incumbent’s Party (%)	7.3	7.5
N	(885)	(186)
Challenger’s Party (%)	12.1	9.7
N	(560)	(207)

Source.—Public Policy Institute of California Statewide Survey.

Note.—Cell entries are the percentage of respondents identifying with each party who expected to vote against their party affiliation. Partisan registration and identification include leaning independents.

Conclusion

Existing evidence suggests that featuring candidate names in congressional vote intention questions inflates support for incumbents. We argue the opposite is also true: emphasis on party labels diminishes the impact of incumbency. The result is an estimate biased in favor of challengers. Questions of this sort are far from unusual in congressional elections research. In fact, the generic congressional ballot—which refers to parties and not candidates—is perhaps the most common gauge of the national partisan vote for the House. Though its accuracy at this national level is mostly sound (but see Erikson and Sigelman 1995, 2000; Moore and Said 1997), it is clear that the generic ballot does not provide accurate estimates of individual races.

Our findings strongly support Box-Steffensmeier, Jacobson, and Grant (2000), who argue that candidate names encourage votes for the incumbent while party labels promote a more balanced—but not necessarily accurate—estimate. Researchers should take care to include candidate names when measuring vote choice at the district level because party labels alone will usually understate the incumbent’s support. Moreover, any question that does not include candidate names should not be used to analyze district-level variation in vote choice, for it is likely to miss a substantial portion of that variance. On election day, voters in low-key races tend to give higher support to the incumbent, but the generic ballot encourages the same partisan response whether the race is close or a foregone conclusion.

Appendix: The PPIC Survey

QUESTIONS

House vote:

Next, if the election for U.S. House of Representatives were being held today, would you vote for the Republican Party’s candidate or the Democratic Party’s candidate for the House in your district?

Party registration:

On another topic, some people are registered to vote and others are not. Are you absolutely certain you are registered to vote? (If yes: are you registered as a Democrat, a Republican, another party, or as an independent?)

Independent leaners:

(Asked of Independents only) Do you think of yourself as closer to the Democratic Party or the Republican Party?

1.

The problem is serious enough that the National Election Study (NES) attempted to address it in its 1984 and 2000 surveys.

2.

The question wording has varied some over time, but the basic structure has remained the same.

3.

The samples were drawn using the random digit dial (RDD) technique and “last birthday” adult household member selection, which ensured that the data were geographically representative and that they included both listed and unlisted telephone numbers. RDD sampling addresses the issue of the National Election Study’s cluster sampling design that can produce biased samples of individual districts (Stoker and Bowers 2000). There was incomplete documentation of final disposition codes for these surveys by the field service employed to carry them out. Many cases of unknown eligibility remained in the pool of numbers selected for calling. From what can be ascertained from the call records, using the most conservative outcome rate definitions, the AAPOR RR1 response rate is estimated at 9.8 percent and the AAPOR COOP1 cooperation rate is estimated at 27 percent. The data were statistically weighted to be geographically and demographically representative (see Baldassare 2000a, 2000b).

4.

The commercial data set we used to merge in the district numbers (ZipInfo.com 2000) contained a variable indicating the percentage of each zip code that fell within a given congressional district. We used the 95 percent confidence standard to keep only those cases where at least 95 percent of the zip code fell in one congressional district. Using only those cases with 100 percent accuracy did not change the results.

5.

The PPIC Statewide Survey asks respondents for their partisan registration, not their more general partisan affiliation at the time of the survey, so the question is meant to be an estimate of the secretary of state’s registration numbers (see the appendix for details).

6.

We could have made the distinction using the election outcome: for instance, by defining competitiveness as any challenger who received more than 45 percent of the vote. We chose to use money because it more closely identifies the possible explanation behind the results we find. Using money, we can plausibly claim that the results are due to the availability of information. Challengers who spend more money are probably doing a better job of getting their names to the public. Using election outcomes, that causal link would be somewhat more ambiguous.

7.

The hard-fought races were in districts 10, 20, 23, 27, 36, 38, 42, and 49. We tried basing our cutoff on the “hot races” as defined by the California Journal, a periodical covering the state’s politics. We also tried using the challenger’s political experience: those with previous experience in elected office were considered competitive. Both produced the same substantive result.

8.

There appears to be a tendency for the vote to drift over time toward the challenger in hard-fought races, so that the vote for the incumbent is slightly overreported in September and underreported in October, but the change is small, and there are not enough cases in the hard-fought races in each survey wave for this difference to achieve statistical significance.

9.

We have dropped independents from this table. The number of “pure” independents—independents who do not lean toward one party or the other—is small enough in the PPIC sample that there are not enough cases to analyze these respondents in competitive districts.