Abstract
Public deliberation on the Internet is a promising but unproven practice. Online deliberation can engage large numbers of citizens at relatively low cost, but it is unclear whether such programs have substantial civic impact. One factor in determining their effectiveness may be the communicative features of the online setting in which they occur. Within a Media Richness Theory framework, we conducted a quasiexperiment to assess the civic outcomes of interventions executed online by nonprofit organizations prior to the 2012 U.S. presidential election. The results assess the impact of these interventions on issue knowledge and civic attitudes. Comparisons of the interventions illustrate the importance of considering media richness online, and our discussion considers the theoretical and practical implications of these findings.
Democratic self-government requires constant and complex interactions among diverse populations, which make communication theories essential in understanding its operation. Such theories have particular significance in many Western countries, in which citizens question the efficacy of their own actions, vote at low rates, and exhibit low levels of political knowledge (Dalton, 2008; Delli Carpini & Keeter, 1993). Though skepticism is a hallmark of democracy, declining indicators of institutional legitimacy compromise the very functioning of democratic institutions and point to the need for more efficacious civic interventions (Levine, 2013).
One popular approach holds that stronger deliberative norms and institutions could reduce the dysfunction in modern democratic politics (Gutmann & Thompson, 2004; Parkinson & Mansbridge, 2012). Deliberative theory foregrounds the discursive legitimization of self-government. Citizens can forge and demonstrate collective assent by enacting deliberative discourse norms in meaningful settings, such as juries, forums, or newly crafted public spaces (Dryzek, 2010; Fishkin, 2009; Parkinson & Mansbridge, 2012). In this view, stronger discursive ecologies can produce more legitimate democracies.
A crucial empirical question remains, however. Researchers have begun to investigate deliberation's ability to educate and inspire those citizens who avail themselves of deliberative opportunities (Mendelberg, 2002; Pincock, 2012), but the more crucial question is whether deliberation's civic benefits can reach the broader population. The scalability of online deliberative formats has inspired civic reformers (Davies & Chandler, 2012), despite early concerns that the loss of face-to-face proximity would diminish the deliberative experience in online settings (e.g., Barber, 1984; Gastil, 2000). Quasideliberative interactions now take place regularly in a variety of online discussion contexts (Black, Welser, Cosley, & DeGroot, 2011; Dahlgren, 2005; Dahlberg, 2011; Lewiński & Mohammed, 2012; Wright & Street, 2007). Though researchers have investigated online versus face-to-face modalities (e.g., Baek, Wojcieszak, & Delli Carpini, 2012), few studies have examined directly the variations in communication modalities used for online civic intervention.
In this essay, we compare the efficacy of different online designs created and executed by a partnership of two nongovernmental organizations, AmericaSpeaks1 and Face the Facts USA. We report the results of a 2012 field experiment that compares the effects of different computer-mediated civic interventions on participants' issue knowledge and deliberative attitudes. Before turning to those results, we begin by framing this study within the broader framework of Media Richness Theory, which conceptualizes a “rich” communication environment as one that involves natural language, multiple communication channels, synchronous feedback, and personal focus. Following this theory, we aim to show that variations in the richness of an online medium can explain the relative impact of different methods of online civic intervention.
Deliberation and Online Media Richness
The particular context of our research permits empirically testing broader theoretical propositions about variability in online deliberative design and the resultant effects on individuals' knowledge and civic attitudes. We begin with a discussion of these two subjects to advance hypotheses about the impact of different online civic interventions.
The Civic Impact of Deliberating
Democratic deliberation denotes a decision-making process that meets high standards for both analytic rigor and democratic social relations (Burkhalter, Gastil, & Kelshaw, 2002), but “deliberativeness” may also be a variable used to characterize particular discursive episodes and settings (Gastil, 2008).
As a descriptor for political practices, deliberation references both formal processes with elaborate structural designs (Black, Burkhalter, Gastil, & Stromer-Galley, 2010) and discussions that do not make decisions, per se, but seek to improve individual judgments (e.g., Fishkin, 2009). “Deliberative” also describes a political environment where critical-rational argumentation and social norms like civility and inclusivity are simultaneously privileged as the standards for decisional legitimacy (Gutmann & Thompson, 2004). Thus, although noninteractive civic interventions cannot be conceived of as deliberative in the former episodic-appraisal sense, they can contribute to deliberative norms more generally by spurring reflective processes and deliberative habits (Goodin, 2003).
Democratic theory posits that participating in even quasi-deliberative activities can improve one's issue knowledge, reinforce faith in democracy, and inspire one to participate in public life (Habermas, 1996; Pateman, 1970). Deliberation can shape participants' knowledge on specific issues under discussion, raise awareness of their own and others' interests and values, foster a broader understanding of the public good, and increase political efficacy (Burkhalter et al., 2002; Delli Carpini et al., 2004; Pincock, 2012; Steiner, 2012). Deliberative participation can even increase confidence in deliberation itself (Burkhalter et al., 2002; Knobloch & Gastil, 2014).
In testing the efficacy of different modes of online deliberation, we distinguish between deliberation's potential impact on policy knowledge versus civic attitudes. First, issues forums and similar events have increased public knowledge on policy facts (Farrar et al., 2010; Fishkin, 2009; Pincock, 2012) and the relevant arguments in policy debates (Cappella, Price, & Nir, 2002). Second, deliberative participation can provide a sense of political empowerment, influence perceptions of governmental responsiveness, and boost one's willingness to participate in civic life (Delli Carpini et al., 2004; ; Morrell, 2010; Nabatchi, 2010). Participation in deliberation has also been shown to increase individuals' faith in the deliberative process (Knobloch & Gastil, 2014). Such attitudinal outcomes differ conceptually from knowledge gains, and that difference will be important to consider when theorizing the impact of varied modes of online deliberative intervention.
The Civic Utility of Variable Media Richness
Some deliberative interventions outpace others in achieving desired civic outcomes, and the nature and magnitude of their impact depends on program design features. Though they have much in common, face-to-face civic engagement programs vary in precisely this way: From Deliberative Polls to Citizens' Juries to participatory budgeting, each differentiates itself from the others in its precise purpose, design, and outcomes (Fung, 2007; Nabatchi et al., 2010).
Likewise, “online deliberation” encompasses many forms and sites of computer-mediated political discourse, such as weblogs, chat rooms, message boards, forums, online news media, and governmental websites (Coleman & Moss, 2012; Davies & Chandler, 2012; Towne & Herbsleb, 2012). Whereas deliberation scholars regard the Internet as a generally promising discursive public space (Dahlgren, 2005; Freelon, 2010; Gerhards & Schäfer, 2010), every online environment involves complex design choices (Davies & Chandler, 2012). Thus, Wright and Street (2007) argue that experimental testing of design features “may be crucial to appreciating the democratic potential of the web” (p. 864).
Design choices concerning participant interaction are key because a basic deliberative discourse feature is the opportunity for participants to “exchange opinions as well as incorporate and respond to others' viewpoints” (Wilhelm, 2000, p. 88). With this in mind, researchers have contrasted face-to-face versus computer-mediated interaction designs (Baek et al., 2012; King, Hartzel, Schilhavy, Melone, & McGuire, 2010; Monnoyer-Smith & Wojcik, 2012; Pedrini, 2014; Tucey, 2010) and explored variations in online formats (Davies & Chandler, 2012; Shaw & Benkler, 2012; Towne & Herbsleb, 2012).
Media richness theory (Daft & Lengel, 1986) provides a particularly useful framework for differentiating civic educational interventions. The theory was developed to explain the varying impacts observable between impersonal and limited media, such as typed text interfaces, versus more intimate media that can facilitate complex information along many channels at once, such as in video conferencing. Much work has been done specifying specific concepts in the theory, such as task efficacy (e.g., “decision quality” as discussed by Kahai & Cooper, 2003) and the operational distinction between complexity and equivocality (Sheer & Chen, 2004), yet empirical support for the greater efficacy of richer media has been weak (Dennis, 2009).
In the context of online deliberation, however, the theory likely has utility (Davies & Chandler, 2012). As Ferber et al. (2007) argue, research on the efficacy of online political participation requires contrasting different designs and venues, “and such an assessment will be more meaningful if the examination occurs within some common framework” (p. 399). Thus, we adopt Media Richness Theory's criteria to distinguish among levels of differentially rich communication: (a) natural language use, (b) multiplicity of communicative cues (e.g., nonverbal cues, vocal inflection), (c) synchronous feedback, and (d) personal focus, in terms of the recipient (e.g., Sheer & Chen, 2004, p. 77).
Our empirical investigation maps these communication features onto different civic education interventions. Broadly speaking, richer media have greater potential for democratic deliberation, which simultaneously requires both complex information processing on matters of public controversy and the maintenance of democratic social norms (Burkhalter et al., 2002). As Sheer observes, “Rich media have greater personal information-carrying capacity… and, thus, facilitate interpersonal communication and relationship development” (2011, p. 83). That capacity may better serve the purpose of maintaining interpersonal respect during the substantive disagreement necessary for deliberation (Barber, 1984). Also, in comparison with tasks based on straightforward factual information, deliberation on matters of public debate requires processing information while making credibility judgments about the speakers and sources providing information (van Gelder, 2012). Richer media should better serve that social purpose, because of the utility of additional social information, such as feedback and cue multiplicity for ambiguous communicative tasks (see Solomon & McLaren, 2008).
Tying this back to the two dimensions of impact—knowledge gains and civic attitude shifts (Pincock, 2012)—the relational advantage of rich media may prove more important to shifting one's attitudes about fellow citizens, political life, and the efficacy of deliberation itself. By contrast, learning relevant information may occur more readily in relatively lean media, which focus on key facts rather than the social environment in which those are delivered.
Hypotheses
Working within this general model, we integrate the preceding literatures on deliberative outcomes and media richness to advance hypotheses about the differential effects of variably media-rich civic interventions for policy knowledge and civic attitudes. Though our central purpose is contrasting different intervention methods, we begin by making explicit the null hypothesis, which predicts that civic interventions have no impact. This view of an overtaxed and apathetic public—unresponsive to entreaties to deliberation—has adherents (Hibbing & Theiss-Morse, 2002), though considerable evidence has accumulated to the contrary (Neblo, Esterling, Kennedy, Lazer, & Sokhey, 2010). Thus, we begin with a single hypothesis that extends across all the dependent variables and generalizes across the various online formats.
H1: Involvement in a deliberative civic intervention will lead to greater factual knowledge and more favorable attitudes toward civic engagement than assignment to a control group.
Beyond that general prediction, we anticipate variation in the effects of the different civic engagement interventions. Considerable evidence shows the differential effects of varied communication modes and degrees of interactivity in noncivic educational settings (Mayer, 2003; Moreno, 2006), so deliberative educational opportunities are likely to differ in their impact based on their design (Davies & Chandler, 2012).
The first difference we predict among deliberative online methods concerns knowledge. The Deliberative Poll's advocates emphasize knowledge gains as a central deliberative output (Fishkin, 2009), but it is not clear that such effects benefit from social interaction. Less rich formats that provide fixed information to participants (without the distraction of discussion) could prove the most effective means of increasing knowledge relevant to the policy issue under discussion (Davies & Chandler, 2012).
H2: Participation in leaner online interventions will lead to significantly higher factual knowledge retention compared to richer interventions.
Although knowledge gains may be highest in formats without interactive richness attributes, we expect more dialogic forms of deliberation to foster greater political empowerment and thereby change participants' convictions about the efficacy of deliberation itself (Knobloch & Gastil, 2014). We test this broader claim across four different civic attitudes. First, we anticipate that interaction can boost confidence in one's knowledge in the relevant issue domain addressed by a given online deliberation. This concept of “political information efficacy” was conceived and tested in the context of viewing national Presidential debates (Kaid & Postelnicu, 2005), so unidirectional media may be sufficient to yield an increase. More interactive formats, however, provide users increasingly effective means of bolstering their confidence in their relevant knowledge. Only the more interactive formats afford participants the chance to ask those questions to which they do not yet know answers.
H3a: Participation in richer online interventions will lead to significantly higher political information efficacy compared to leaner interventions.
Richer formats have a greater advantage when one turns toward more abstract conceptions of political efficacy. Intensive deliberation can boost one's sense of external efficacy—trust in the responsiveness of the larger political system—by drawing participants from a passive mode into a more engaged approach to citizenship (Nabatchi, 2010; Pincock, 2012). If faith in the political process comes from interaction with fellow citizens, such an effect should be found when communication media are sufficiently rich to support synchronous individual deliberation. By contrast, passive interventions that do not engage participants in deliberation directly should have little impact on this attitude.
H3b: Participation in richer online interventions will lead to significantly higher external political efficacy compared to leaner interventions.
Theories of group deliberation also have posited that participating in highly interactive political discussions should yield more support for the deliberative process, owing to the generally rewarding experience of deliberation (Burkhalter et al., 2002). This claim has found support in studies of face-to-face groups, but no support in very lean online interaction environments (Knobloch & Gastil, 2014). Deliberating online could enhance one's general faith in the efficacy of using highly interactive deliberation to arrive at public judgment. Because richer media employ such processes, it follows that richer online experiences will shape this attitude most strongly.
H3c: Participation in richer online interventions will lead to significantly higher faith in deliberation compared to leaner interventions.
A final attitude of interest concerns one's appreciation of the need for fact-based discussion. Many deliberative formats, such as the Deliberative Poll (Fishkin, 2009) and the Citizens' Initiative Review (Gastil, 2014), emphasize factual content that can shape policy judgments. All the online discussion formats we examine place factual information at the center. The leaner media formats can provide factual information most bluntly, as evidenced by Daft and Lengel's (1986) identification of “numeric documents”—or pure data—as the least rich medium (p. 560). By contrast, richer formats give participants an increasing opportunity to reflect on, contest, and appreciate the facts that come up during discussion. Though the simplest modes might yield the greatest knowledge gains, the more interactive opportunities appear better suited to foster an appreciation for factual knowledge in public debate. In the marketing domain, the enhanced interactivity of online communities has been found to be associated with increased appreciation for factual information about products (Archer-Brown, Piercy, & Joinson, 2013). We expect the same to be true of policy information.
H3d: Participation in richer online interventions will lead to significantly higher assessments of the value of fact-based discussion compared to leaner interventions.
Method
Participants
The data we present come from a quasiexperimental design. Study participants were recruited to take part in one of four online programs conducted between 11 October and 5 November 2012 by the civic engagement organizations AmericaSpeaks and Face the Facts USA. Participants were solicited via Amazon Mechanical Turk (AMT), a web-based task marketplace platform that has become increasingly common in academic research because of its low cost and demographic diversity. Instructions for participation were listed as tasks on the AMT marketplace.
AMT users became study participants by accepting posted tasks, completing the experimental intervention, then submitting an online questionnaire. There were 395 participants in the control group, 252 in the text group, 353 in the video group, 138 in the webcast group, and 112 in the deliberative video chat group. Unique participant ID numbers associated with the AMT accounts allowed tracking of participants through the interventions, as well as identification and deletion of repeat participants. Participants were compensated with $1.00 for the control, video, and text conditions, $6 for attending the Spreecast and completing the questionnaire, and $8 for participating in the Hangout and completing the questionnaire. Of the 1,124 cases who provided demographic information, 537 were female (48.5%) and 903 were Caucasian (81.7%). The sample included 205 Republicans (18.7%) and 527 Democrats (48.0%).2 The average age was 33 years (SD = 11.1).
Procedure & Instrumentation
After accepting an assignment, the AMT recruits received instructions and gave informed consent through the Qualtrics web survey interface. Participants in interactive conditions were sent to participate in programs convened by the program hosts in their native ecology. Participants in the static conditions were exposed to videos and textual statements taken from the Face the Facts USA website and embedded in the questionnaire. All participants completed follow-up questionnaires. The following sections describe the treatment conditions, which map onto the indicators of media richness as well as possible, given that our quasiexperimental design employed preset online formats actively in use. Table 1 shows the five conditions that produced the quasiexperimental data analyzed in this article. Each of the interventions represents a progressively richer medium.
Conceptual Mapping of Media Richness to Civic Interventions
| Richness Indicators | ||||
|---|---|---|---|---|
| Observed Civic Intervention | Natural Language | Cue Multiplicity | Synchronous Feedback | Personal Focus |
| Control | ||||
| Text | √ | |||
| Video | √ | √ | ||
| Webcast | √ | √ | √ | |
| Video Chat | √ | √ | √ | √ |
| Richness Indicators | ||||
|---|---|---|---|---|
| Observed Civic Intervention | Natural Language | Cue Multiplicity | Synchronous Feedback | Personal Focus |
| Control | ||||
| Text | √ | |||
| Video | √ | √ | ||
| Webcast | √ | √ | √ | |
| Video Chat | √ | √ | √ | √ |
Conceptual Mapping of Media Richness to Civic Interventions
| Richness Indicators | ||||
|---|---|---|---|---|
| Observed Civic Intervention | Natural Language | Cue Multiplicity | Synchronous Feedback | Personal Focus |
| Control | ||||
| Text | √ | |||
| Video | √ | √ | ||
| Webcast | √ | √ | √ | |
| Video Chat | √ | √ | √ | √ |
| Richness Indicators | ||||
|---|---|---|---|---|
| Observed Civic Intervention | Natural Language | Cue Multiplicity | Synchronous Feedback | Personal Focus |
| Control | ||||
| Text | √ | |||
| Video | √ | √ | ||
| Webcast | √ | √ | √ | |
| Video Chat | √ | √ | √ | √ |
Control
Some participants were sent directly to the follow-up survey without exposure to any of the executions, thus constituting a baseline group.
Text
Participants sent to the ‘text’ condition viewed two paragraphs of textual content taken from two Face the Facts USA website fact pages. The paragraphs were embedded directly into the online questionnaire after the informed consent screen.
These paragraphs employed natural language insofar as they present data in prose rather than tables. However, the paragraphs were characterized by cue singularity, lack of feedback, and no personal focus.
Video
Participants in this condition viewed two animated videos created by Face the Facts USA to provide specific facts relevant to the 2012 presidential campaign. The videos were from the same content pages as the “text” exposure paragraphs. The videos were embedded directly into the online questionnaire, and participants were exposed to them immediately after completing informed consent.
These videos employed natural language and cue multiplicity, including dynamic visuals and narration. However, the videos were characterized by lack of feedback and no personal focus.
Webcast
Participants in the webcast condition were instructed to register for a live webcast and arrive at a URL at a designated time. Using the http://Spreecast.com webcasting platform, Face the Facts USA conducted a program to discuss political issues relevant to the 2012 presidential campaign. The program was largely preplanned and consisted of a discussion of factual information3 and an interview with an expert. It was semi-interactive, however, insofar as participants were able to submit questions to the host and receive responses in real time, and use a chat interface to communicate anonymously with other participants. Each webcast lasted 30 minutes, after which participants were asked to complete the online questionnaire.
These webcasts employed natural language, cue multiplicity, and a feedback mechanism. However, the anonymity of viewers and their limited interaction channel did not offer personal focus. It is important to note that the concept of ‘immediate feedback’ in Media Richness Theory is nuanced (see Dennis & Kinney, 1998, p. 260). The feedback capacity supported by the webcast intervention was quite poor; this feature differentiates the webcast from the interactive video chat condition.
Video Chat
Participants in the video chat condition were instructed to register for a small-group discussion section and arrive at a URL at a designated time. The groups included four to eight participants. Using Google Hangout technology, AmericaSpeaks facilitators led discussion using a topic guide on jobs and the economy. The session consisted of participant introductions, a factual background presentation,4 and a facilitated discussion among the participants about their values, perspectives, and opinions. It concluded by considering actions that might be taken to address national economic challenges. Each session lasted about one hour, after which participants completed the online questionnaire.
These video chat sessions featured natural discourse language, cue multiplicity supported by webcams with synchronous audio, a rich feedback potential,5 and personalization because participants were able to identifiably and personally guide the conversation.
Dependent Measures
Five outcomes were analyzed to measure factual knowledge and civic attitudes, which constitute key outcomes identified in empirical theories of deliberation (Burkhalter et al., 2002; Pincock, 2012). With the exception of the Factual Knowledge Index, the dependent variables were scales composed of averaged items measured on a 7-point Likert-type agree/disagree scale.
The Factual Knowledge Index was measured with items based on the factual information content of the experience. The factual materials were available to (but not the emphasis of) the Hangout discussions, were reviewed during Spreecasts, and were incorporated in the textual and video exposures. The factual knowledge items measured facts to which the particular group was exposed. Because the unidirectional exposures were solely informational, respondents in these groups were asked for point responses (e.g., “In 2010, what was the average monthly Social Security payment?”), whereas those in interactive conditions were asked for relative responses (e.g., “Between 2009 and 2011, employment among FOREIGN-BORN workers rose by 5.2 percent. Is this higher or lower than the job recovery for U.S.-BORN workers?”). Two questions were asked per exposure, so index scores were 0 (none correct), .5 (one correct), and 1 (both correct).
Political Information Efficacy (PIE) (Kaid, McKinney, and Tedesco, 2007; Kaid & Postelnicu, 2005) was measured using a 4-item scale (Cronbach's Alpha = .87) that assessed the degree to which participants believed they could competently employ their political knowledge and information to successfully participate as citizens (e.g., “I feel that I have a pretty good understanding of the important political issues facing our country.”).
External Efficacy was a 4-item scale (α = .75) measuring the participant's belief that the government is responsive to the public (e.g., “Under our form of government, the people have the final say about how the country is run, no matter who is in office.”).
Faith in Deliberation is a 3-item scale (α = .70) measuring the participant's belief that discursive decision-making is a realistic and productive form of political interaction (e.g., “Even people who strongly disagree can make sound decisions if they sit down and talk.”).
Value Factual Discourse is a 3-item scale (α = .69) measuring the participant's belief in the imperativeness of factual quality discourse (e.g., “When we hear claims that are false or misleading, we have a responsibility to speak up and CORRECT them.”).
Results
Our use of quasiexperimental data raises the need for caution because random assignment was not assured. Variations in sample characteristics pose a potential inferential threat (Campbell & Stanley, 1963, p. 36). Prior to analysis, we conducted a randomization check of demographic means and frequencies across each study group and found no differences in education, income, age, ethnicity, political party identification, and political interest; however, sex was not evenly distributed (χ2 = 10.448, p = .034). Standardized residuals indicated that more males participated in the control condition than expected, whereas more females participated in the video condition. The following analyses were thus conducted as ANCOVAs to control for the variation in sex between conditions. The pattern of results was not altered, so the ANOVA main-effects are reported.
The impact of media richness, operationalized by the civic intervention conditions, was assessed by one-way ANOVA and post hoc tests at alpha level .05. Table 2 reports LSD comparisons of adjusted group means across the five dependent variables of interest.
ANOVA Comparisons of Dependent Variable Means for Experimental Conditions
| Condition | |||||||
|---|---|---|---|---|---|---|---|
| Dependent Variable | Control | Text | Video | Webcast | Video Chat | F | ηp2 |
| Knowledge index (KI) | .261a | .573b | .740c | .469d | .262a | 102.398** | .270 |
| (.29) | (.37) | (.32) | (.37) | (.33) | |||
| Political info. efficacy (PIE) | 4.770a | 4.944b | 5.010b | 5.093b | 4.877ab | 2.439* | .009 |
| (1.28) | (1.24) | (1.13) | (1.29) | (1.56) | |||
| External efficacy (EE) | 3.616a | 3.761ab | 3.857b | 3.909b | 3.996b | 3.097* | .011 |
| (1.09) | (1.19) | (1.23) | (1.31) | (1.17) | |||
| Faith in delib. (FID) | 5.689ab | 5.579a | 5.699ab | 5.751b | 6.011c | 3.167* | .011 |
| (.87) | (.97) | (.85) | (.82) | (.81) | |||
| Value factual disc. (VFD) | 5.976 | 5.891 | 6.020 | 6.029 | 5.952 | .815 | .003 |
| (.83) | (1.02) | (.87) | (.88) | (1.00) | |||
| Condition | |||||||
|---|---|---|---|---|---|---|---|
| Dependent Variable | Control | Text | Video | Webcast | Video Chat | F | ηp2 |
| Knowledge index (KI) | .261a | .573b | .740c | .469d | .262a | 102.398** | .270 |
| (.29) | (.37) | (.32) | (.37) | (.33) | |||
| Political info. efficacy (PIE) | 4.770a | 4.944b | 5.010b | 5.093b | 4.877ab | 2.439* | .009 |
| (1.28) | (1.24) | (1.13) | (1.29) | (1.56) | |||
| External efficacy (EE) | 3.616a | 3.761ab | 3.857b | 3.909b | 3.996b | 3.097* | .011 |
| (1.09) | (1.19) | (1.23) | (1.31) | (1.17) | |||
| Faith in delib. (FID) | 5.689ab | 5.579a | 5.699ab | 5.751b | 6.011c | 3.167* | .011 |
| (.87) | (.97) | (.85) | (.82) | (.81) | |||
| Value factual disc. (VFD) | 5.976 | 5.891 | 6.020 | 6.029 | 5.952 | .815 | .003 |
| (.83) | (1.02) | (.87) | (.88) | (1.00) | |||
Note.
p < .05,
p < .01.
Standard deviations appear in parentheses. Means with differing subscripts within rows are significantly different at the p < .05 based on Fisher's Least Significant Difference (LSD) comparisons.
ANOVA Comparisons of Dependent Variable Means for Experimental Conditions
| Condition | |||||||
|---|---|---|---|---|---|---|---|
| Dependent Variable | Control | Text | Video | Webcast | Video Chat | F | ηp2 |
| Knowledge index (KI) | .261a | .573b | .740c | .469d | .262a | 102.398** | .270 |
| (.29) | (.37) | (.32) | (.37) | (.33) | |||
| Political info. efficacy (PIE) | 4.770a | 4.944b | 5.010b | 5.093b | 4.877ab | 2.439* | .009 |
| (1.28) | (1.24) | (1.13) | (1.29) | (1.56) | |||
| External efficacy (EE) | 3.616a | 3.761ab | 3.857b | 3.909b | 3.996b | 3.097* | .011 |
| (1.09) | (1.19) | (1.23) | (1.31) | (1.17) | |||
| Faith in delib. (FID) | 5.689ab | 5.579a | 5.699ab | 5.751b | 6.011c | 3.167* | .011 |
| (.87) | (.97) | (.85) | (.82) | (.81) | |||
| Value factual disc. (VFD) | 5.976 | 5.891 | 6.020 | 6.029 | 5.952 | .815 | .003 |
| (.83) | (1.02) | (.87) | (.88) | (1.00) | |||
| Condition | |||||||
|---|---|---|---|---|---|---|---|
| Dependent Variable | Control | Text | Video | Webcast | Video Chat | F | ηp2 |
| Knowledge index (KI) | .261a | .573b | .740c | .469d | .262a | 102.398** | .270 |
| (.29) | (.37) | (.32) | (.37) | (.33) | |||
| Political info. efficacy (PIE) | 4.770a | 4.944b | 5.010b | 5.093b | 4.877ab | 2.439* | .009 |
| (1.28) | (1.24) | (1.13) | (1.29) | (1.56) | |||
| External efficacy (EE) | 3.616a | 3.761ab | 3.857b | 3.909b | 3.996b | 3.097* | .011 |
| (1.09) | (1.19) | (1.23) | (1.31) | (1.17) | |||
| Faith in delib. (FID) | 5.689ab | 5.579a | 5.699ab | 5.751b | 6.011c | 3.167* | .011 |
| (.87) | (.97) | (.85) | (.82) | (.81) | |||
| Value factual disc. (VFD) | 5.976 | 5.891 | 6.020 | 6.029 | 5.952 | .815 | .003 |
| (.83) | (1.02) | (.87) | (.88) | (1.00) | |||
Note.
p < .05,
p < .01.
Standard deviations appear in parentheses. Means with differing subscripts within rows are significantly different at the p < .05 based on Fisher's Least Significant Difference (LSD) comparisons.
To test the overall hypothesis that the conditions together represent an effect of civic intervention (H1), planned contrasts between the control group and the four intervention conditions for all five dependent variables were conducted. The contrasts were significantly different for the knowledge index (KI) (t1109 = 11.27, p ≤ .001), political information efficacy (PIE) (t451.6 = 2.34, p = .02), and external efficacy (EE) (t587 = 3.43, p ≤ .001). The contrasts were not significantly different for outcome scales indicating faith in deliberation (FID) (t1119 = 1.19, p = .23) or value of factual discourse (VFD) (t501.3 = −.043, p = .967). Thus, H1 was partially supported.
To test the knowledge retention hypotheses (H2), the mean Knowledge Index scores were compared. The omnibus test was significant, F4,1109 = 102.398, p < .01, ηp2 = .27. The LSD mean comparisons showed that knowledge recall for Video Chat was not significantly different from the control group, the Webcast was significantly higher than the control group, the Text group was significantly higher than the Webcast, and the Video was significantly higher than the Text.
To test the political information efficacy hypothesis (H3a), mean PIE scores were compared. The omnibus test was significant, F4,1119 = 2.439, p < .05, ηp2= . 009. The LSD mean comparisons showed that while all but the Video Chat are significantly higher than the control group, there was no significant difference between the comparison conditions.
To test the external efficacy hypothesis (H3b), mean EE scores were compared. The omnibus test was significant, F4,1119 = 3.097, p < .05, ηp2 = .011. The LSD mean comparisons showed that while all but the Text were significantly higher than the control group, there is no significant difference between the comparison conditions.
To test the faith in deliberative processes hypothesis (H3c), mean FID scores were compared between the five conditions. The omnibus test was significant, F4,1119 = 3.167, p < .05, ηp2 = .011. The LSD mean comparisons showed that while the Text, Video, and Webcast means were not significantly different from the control, the Video Chat was significantly higher than the other conditions. The Webcast mean was significantly higher than the text condition but not the video condition, lending partial support to the hypothesis that the richer experiences would lead to higher FID than the leaner experiences. Thus, H4 was partially supported.
To test the factual discourse hypothesis (H3d), mean VFD scores were compared between the five conditions. The omnibus test was not significant, F4,1119 = 0.815, p > .05, ηp2 = .003. H5 is not supported.
Discussion
Our quasiexperimental study shed some light on the likely outcomes of deliberative civic engagement online. We directed participants from a general web population to unmodified intervention experiences of varying richness executed by not-for-profit organizations, then collected relevant outcome measures. The data show that deliberative interventions' net impact varies across levels of richness and for factual knowledge versus civic attitudes.
The interventions, in general, showed a net impact relative to the control group, but among such online discussion formats, we had hypothesized that richer models would prove more powerful. A key exception is our finding that even though (or perhaps because) the highly involved facilitated video-chat intervention lasted substantially longer than the simpler exposures, it could not produce knowledge gains relative to the control. This is problematic because high-quality deliberation demands a well-informed deliberative body. The lean unidirectional modalities, particularly video, were effective in producing knowledge gains, consistent with previous research (Davies & Chandler, 2012).
In practical terms, this suggests that the designers of deliberative online civic engagement venues might do well to offer mixed modalities to meet different knowledge goals at different points in a deliberative process. For example, a civic organization might design a relatively brief online deliberation as a two-stage process. The initial stage would employ text-based and one-way video media to help participants absorb relevant information. A subsequent stage would feature more interactive media such as facilitated video chat to enable characteristically deliberative conduct such as the development of social ties and mutual understanding among participants, reflection on values as well as facts, formulation of persuasive arguments, and negotiation of trade-offs required by different policy choices (Gastil, 2008). During this second stage textual and one-way video resources—the presence of which could be highlighted by facilitators (Polletta, Chen, & Anderson, 2008)—would remain available for use in refreshing participants' knowledge and as evidence to inform arguments. This design could be elaborated for a lengthier, multiday or multiweek online deliberative process on a complex policy issue, such as those addressed by Consensus Conferences (Hendriks, 2005) and Citizens' Assemblies (Warren & Pearse, 2008). An online session could be devoted to each component of the issue, with a concluding session focused on integration of previous discussions as well as final decision making, and each session would have the two-stage structure described above. Examples of cases in which this elaborated design might be applied include future versions of the What's Next California? Deliberative Poll (Smith, 2011) and the Australian Citizens' Parliament (Carson, Gastil, Hartz-Karp, & Lubensky, 2013).
Though our data do not suggest internal and external efficacy vary as a function of interactivity, we did find that more interactive engagements affect faith in the efficacy of group deliberative processes. This is consistent with the view in deliberative theory that citizens ascribe legitimacy to collective governmental processes that give them meaningful roles in policy discussions (Burkhalter et al., 2002; Knobloch & Gastil, 2014).
That said, data collected by quasiexperimental methods are always subject to cautious interpretation. We did not have control over the execution of those experiences. This study focused on variability between richness levels without assessing directly any variation within them. Continuation of this investigation across different online civic programs that offer differential richness within-medium (e.g., static video versus video content delivered after interactive input), would provide more data to address the relative magnitudes of within- and between-factor variations.
Regarding our claims about factual knowledge gains, we recognize there are many types of political information beyond policy-relevant facts. Issue-specific factual knowledge enhances citizens' capacity to engage in particular public deliberations. Our conclusions do not extend to other forms of knowledge, such as understanding government functions (e.g., the responsibilities of the branches of government) and keeping track of specific elected officials (Delli Carpini & Keeter, 1993). Moreover, recent work on cognitive diversity in democratic decision making (Landemore, 2013) suggests that group-level diversity of cognitive perspectives may be as important an influence on deliberative outcomes as individual-level factors. The claim of Media Richness Theory that differentially rich media should match task applications of their information content may be a useful concept for this investigation; whereas lean media are more useful for producing factual understanding, the social understanding demanded by deliberative theory (Burkhalter et al., 2002) might be a different type of knowledge we did not capture.
Indeed, we examined our data post hoc to contrast knowledge gains against a single item gauging participants' sense that the intervention “was important for helping me understand the issue.” Moving from the low richness of video to the high richness of the Google Hangout discussions, Figure 1 shows a nearly inverse relationship between richness improving knowledge versus aiding in deeper understanding. In other words, lean media may aid factual learning, but richer media may provide the context necessary for grasping the public import of such facts.
Standardized Means for Knowledge Index and Helpfulness
Though much is gained by taking a quasiexperimental approach to field studies such as these, future research should include more carefully tuned experimental comparisons of interventions. In particular, our experimental conditions contrast not only media richness but also sheer information density. The text and video exposures were brief bursts of information compared to the webcast and video chat, which could last thirty minutes to an hour. Such variations introduce a potential confound.
In conclusion, this study has responded to Ferber et al.'s (2007) call for “an assessment of electronic deliberation” that looks “at many venues… within some common framework” (p. 399). We used a media richness model to assess a variety of civic engagement interventions executed by not-for-profit organizations in a real-world intervention during an election. Our findings indicate that knowledge and prodeliberative attitudes may be acquired through participation in online deliberations, but interactivity was not a deliberative panacea. Richer media, characterized by higher levels of interactivity, encouraged participants to believe in the deliberative process, but interferes with the process of learning straightforward policy-relevant information.
Research has progressed from attempting to characterize the Internet as more or less deliberative than face-to-face meetings to investigating the particular features of the online context. Our findings suggest that interactivity is not necessarily a more deliberative feature in itself. Rather, interactivity should be conceptualized as a trait of an interactive modality that may be matched to particular deliberative purposes. Just as jurors move from courtroom to jury room, so too might online deliberation formats do well to mix opportunities for free-flowing interaction with more limited settings that emphasize the learning of salient factual information. Such a mix might best realize the multifaceted goals of deliberative theory.
Notes
AmericaSpeaks (1995–2014) developed public meeting designs that wedded digital technology to large-scale face-to-face meetings. This project, which was among the last it organized, attempted to move issues-based deliberation and civic education into a variety of online settings.
Ideally, the discussions would have had a greater ideological balance, which is the goal in processes that employ random samples (e.g., Fishkin, 2009). That said, it is not uncommon for public issues forums to attract an ideologically imbalanced population and still yield meaningful civic educational outcomes (e.g., Gastil & Dillard, 1999).
The factual information was presented as a discussion of the Face the Facts materials, but those video and text materials were summarized orally, rather than shown as they were in the exposure conditions.
The factual background presentation was based on the information produced by Face the Facts used for the text and video. However, the text and video exposures did not appear verbatim as part of the presentation.
But see Sheer (2011, p. 84) on computer-mediated communication interaction capacity.
References
About the Authors
David L. Brinker, Jr. (dlb385@psu.edu) is a Ph.D. candidate at the Pennsylvania State University. His research interests are interpersonal communication, political communication, and public deliberation. Address: Department of Communication Arts and Sciences, Pennsylvania State University, 316 Sparks Building, University Park, Pennsylvania, 16802.
John Gastil (jgastil@psu.edu) is a professor in the Department of Communication Arts and Sciences at the Pennsylvania State University, where he also directs the McCourtney Institute for Democracy. Gastil studies group decision making, public deliberation, and political communication. Address: Department of Communication Arts and Sciences, Pennsylvania State University, 240H Sparks Building, University Park, Pennsylvania, 16802.
Robert C. Richards, Jr. (robert.c.richards@psu.edu) is a Ph.D. candidate at the Pennsylvania State University. His research interests are political communication, legal communication, and legal informatics. Address: Department of Communication Arts and Sciences, Pennsylvania State University, 316 Sparks Building, University Park, Pennsylvania, 16802.
