Community notes increase trust in fact-checking on social media

Abstract Community-based fact-checking is a promising approach to fact-check social media content at scale. However, an understanding of whether users trust community fact-checks is missing. Here, we presented n=1,810 Americans with 36 misleading and nonmisleading social media posts and assessed their trust in different types of fact-checking interventions. Participants were randomly assigned to treatments where misleading content was either accompanied by simple (i.e. context-free) misinformation flags in different formats (expert flags or community flags), or by textual “community notes” explaining why the fact-checked post was misleading. Across both sides of the political spectrum, community notes were perceived as significantly more trustworthy than simple misinformation flags. Our results further suggest that the higher trustworthiness primarily stemmed from the context provided in community notes (i.e. fact-checking explanations) rather than generally higher trust towards community fact-checkers. Community notes also improved the identification of misleading posts. In sum, our work implies that context matters in fact-checking and that community notes might be an effective approach to mitigate trust issues with simple misinformation flags.


A. Descriptive statistics
A.1.Dependent variables.Figure S1 shows the distribution of the participants' responses to questions on trust in fact-checks (Figure S1a), misleadingness of posts (Figure S1b), and sharing intentions (Figure S1c) on 7-point Likert scales.A.2. Demographics, beliefs, and CRT.Table S1 shows the frequencies of the responses to questions on demographics and beliefs.
Evidently, the variables are similarly distributed across the experimental conditions.Overall, 51 % of all participants were female, 48 % male, and 1.4 % identify themselves as non-binary.On average, the participants in our study were 42 years old and the majority of all participants did at least attend college (83 %).Out of all participants, 97 % indicated to be native English speakers, 75 % belong to an ethnic majority, and 74 % are vaccinated against COVID-19.Furthermore, more than half of all participants indicated that they believe in God (54 %).Our sample was approximately balanced across political leanings.Overall, 43 % of participants identified as Democrats and 38 % as Republican (19.7 % as third party or other).During the 2020 presidency election, 48 % indicated that they voted for Biden, whereas 45 % voted for Trump.When being forced to choose between Biden an Trump, 52 % prefer Biden as president and 48 % prefer Trump.
Subjects in our study were asked a series of questions regarding their attitude towards risk, trust, and preference for analytical thinking on a 5-point Likert scale.In the median, survey participants were "undecided" on their attitude towards risk (M edian Risk = 3) and trust in democracy (M edianT rustinDem = 3), had "somewhat" trust in general (M edianT rust = 4), and did "not really" prefer doing something that requires little thought over something that is challenging (M edian T hinkingP ref erence = 2).
Subjects in our study were also asked to complete a 4-item Cognitive Reflection Test (CRT).Overall, 43 % passed the CRT, which means that they answered all 4 questions correctly.Notably, this share is comparatively high, pointing towards a high-quality sample of participants from Prolific.Note that these observations align well with expectations.For instance, analysis of voter demographics in various recent US elections revealed that younger people, ethnic minorities, and individuals holding academic degrees were more likely to support the Democratic Party than the Republican Party, while those identifying as religious were less prevalent among Democratic voters (1,2).Similarly, studies on attitudes towards COVID-19 vaccination showed that Democrats were more inclined to get vaccinated (3,4).Moreover, distrust of fact-checkers was shown to be more widespread among pro-Republicans than among pro-Democrats, as pro-Republicans were more likely to perceive fact-checkers as biased (5).A.4. Attrition.Before conducting our analysis, we adjusted the dataset to exclude participants who either did not complete the survey in full, indicated they had responded randomly, engaged in online research during the survey, or indicated they do not use social media.Additionally, those who failed the attention check were removed (see SI, Section D).Table S2 provides an overview of the number of participants excluded on the basis of these criteria.
We find that the variation in participant numbers across different conditions post-cleanup (see Participants (after cleaning) in Table S2) primarily reflects differences in the initial populations of these conditions (see Participants (before cleaning) in Table S2), i.e., randomness in the assignment via the survey tool.The share of participants excluded on the basis of the individual criteria was fairly consistent, and there were no statistically significant differences in the share of participants who completed the survey across the different conditions (Chi-square test: χ 2 = 9; P = 0.2133, two-tailed).Overall, this suggests that attrition did not significantly affect the sizes of the treatment groups in our study.where also asked to rate whether specific post characteristics are important for them when deciding about sharing a post.While accuracy and interestingness were extremely or very important to a majority of participants (91 % and 81 %), it was less important whether a post is funny (41 %) or surprising (22.5 %).Moreover, most participants indicated that it is at least moderately important for them that a post aligns with their beliefs (83 %).The participants' ratings on the importance of specific post characteristics did not differ drastically across different social media platforms and topics.A.6.Perception of fact-checks.Table S4 reports the participants awareness of fact-checks and the influence the fact-checks had on their responses during the survey.Among participants in conditions 2 -4, a vast majority (75 %) was aware of fact-checking prior to participating.However, awareness was substantially higher for expert fact-checks (92 %) than for community flags (65 %) and community notes (66 %).In general, participants' reliance on fact-checks differed a lot, but was similarly distributed across all conditions.Overall, 34 % indicated to have a high or extreme reliance on fact-checks.
Furthermore, participants were asked to indicate what influence the presented fact-check had on their discernment of posts with and without fact-checks during the survey.Overall, the majority of participants stated the the presence of fact-checks made them rate potentially misleading post as less accurate (52 %) or the tag had not influence (20 %).Posts without fact-checks (i.e., that were not potentially misleading) have been rated as more accurate (38.5 %) or the tag had no influence (43 %).Interestingly, the share of participants in condition 4 (community note) indicating that they rated posts with a community note (slightly) more accurate was much higher than in the other conditions (44 % versus 14.5 % and 14.3 %).

B. Estimation results
To predict the participant's (i) trust in fact-checks and (ii) misleadingness ratings, we implemented a hierarchical linear regression model with random intercepts for posts and subjects.Average marginal effects (AMEs) and coefficient estimates are reported in Tables S5 and S7 (trust in fact-checks) and Tables S6 and S8 (misleadingness).The findings are described in detail in Section Results of the main paper.S7.

C. Additional analyses
C.1.Sharing intentions.Note: Participants in our survey also had to state whether they would consider sharing the presented posts on social media.However, research shows that asking about misleadingness before assessing sharing intentions can influence the outcome variable, that is, increase correlation between the identification of misleadingness and sharing intentions (6).The following results should therefore be interpreted with caution.
In our survey, participants assigned to the control condition had significantly lower (t-test: t = −36.11;df = 13395; P < 0.001, two-tailed) sharing intentions (7-point Likert scale normalized to the interval [0, 1]) for misleading (M = 0.13) than for nonmisleading posts (M = 0.23).To quantify intervention effects, we fitted a hierarchical linear regression model with four-way interaction terms and random intercepts for subjects and posts to predict sharing intentions (see Section Methods of the main paper).
Figure S3 shows the average marginal effects (see SI, Table S9 for AMEs).
All fact-checking interventions significantly reduced sharing intentions for misleading posts (see Figure S3a,c,d).Compared to the control condition, sharing intentions for misleading posts were, on average, 2. We also observe some treatment condition effects on untagged non-misleading posts (see Figure S3b).For Biden supporters, In sum, these results suggest that community notes were less successful in reducing sharing intentions for misleading posts than simple misinformation flags.In particular, politically discordant misleading posts with community notes were more likely to be shared (by both Biden and Trump supporters) relative to discordant posts with simple misinformation flags.A potential explanation might be that the community notes themselves were actually perceived as politically concordant (because they were applied to politically discordant original posts).Thus, participants may have been inclined to share the post because they were sharing the community note appended to it, rather than endorsing the original post.(c) AMEs when replacing an expert flag with a community note (i.e., the average difference between the predicted marginal effects for community notes vs. expert flags) across the political leanings of participants (leaning Trump vs. Biden) and the political congruence of the fact-checked posts (concordant, neutral, discordant).(d) AMEs when replacing an expert flag with a community flag (i.e., the average difference between the predicted marginal effects for community flags vs. expert flags) across the political leanings of participants and the fact-checked posts.Control variables and random intercepts for posts and subjects were included.The 95 % confidence intervals (error bars) were derived using the bootstrap method for 1,000 resamples.N = 64,454 observations across 1,810 participants.Full estimation results are in SI, Table S9.

C.2. Demographics and beliefs.
We repeated our regression analyses including information on the participants' demographics and beliefs (see Section Demographics, beliefs, and cognitive reflection in the main paper).This included the participants' gender (male, female, non-binary), level of education (attended college vs. didn't attend college), stance towards God (believes in God vs. doesn't believe in God), ethnicity (ethnic minority vs. not a minority), and COVID-19 vaccination status (vaccinated vs. not vaccinated).In addition, we included the following variables regarding the participants attitude: willingness to take risks (low vs. high), trust in people they interact with in their daily life (low vs. high), trust in democracy (low vs. high), and preference to perform tasks that require thinking (low thinking preference vs. high thinking preference).For all Likert-scale variables, a value of greater than "undecided" (3) was considered high (otherwise low).
Table S10 reports the AMEs for misleadingness ratings under the control condition.We further studied moderation effects, i.e., how the efficacy of fact-checking interventions varied depending on demographics and beliefs.The corresponding AMEs are reported in Tables S11 to S13.

C.3. Reliance on fact-checks.
Participants in the treatment conditions were also asked to indicate whether they tend to rely on fact-checks in general (None (1) to Extreme ( 5)).We repeated our analysis with reliance on fact-checks as an additional explanatory variable (see Section Demographics, beliefs, and cognitive reflection in the main paper).Here, we code responses higher than "Moderate" (3) as a high reliance in fact-checks (otherwise = low).Table S14 reports the AMEs, i.e., how the efficacy of fact-checking interventions varied depending on participants' self-reported reliance on fact-checks across our three dependent variables.We classified four correct answers as "Passed CRT" and less than four correct answers as "Failed CRT".Table S15 reports the average marginal effects (AME) across our three dependent variables.Moderation effects, i.e., how the efficacy of fact-checking interventions varied depending on the outcomes of the CRT are reported in Table S16.The findings are described in detail in Section Demographics, beliefs, and cognitive reflection of the main paper.C.5.Analysis with hierarchical logistic regression models.We repeated our analysis with an alternative model specification using hierarchical logistic regression model and treating the Likert-scale responses as binary variables.Specifically, the dependent variable Trustworthy took the value = 1 if the fact-check was rated at least as somewhat trustworthy (i.e., participants gave a 5 or higher on the 7-point Likert scale) and = 0 otherwise.The dependent variable Misleading took the value = 1 if the fact-check was rated at least as somewhat misleading (i.e., participants gave a 5 or higher on the 7-point Likert scale) and = 0 otherwise.
Table S17 reports the corresponding frequencies across the four experimental conditions.All explanatory variables and random effects specifications were the same as in our main analysis.The logistic mixed-effects models were implemented in R 4.3.2using the glmer package and the marginaleffects package.
Consistent with our main analysis, we observe that users exposed to community notes perceived fact-checks as significantly more trustworthy (see Table S18) than those exposed to simple misinformation flags (all P < 0.01).Furthermore, all fact-checking interventions resulted in participants rating misleading posts as significantly more misleading (see Table S19).In sum, we find that all main findings are robust with similar effect sizes as in our main analysis.• To what extend do you trust the information that comes from the following?(National news organizations, local news organization, friends and family, Social network sites (e.g., Facebook, Twitter), 3rd party fact-checkers (e.g., snopes.com,factcheck.org);each answered on a 5-point likert scale (not at all, a little, a moderate amount, a lot, a great deal)) Depending on the condition participants where assigned to (condition 2-4), we also asked questions about their awareness of the specific type of fact-checks and what influence the respective fact-checks had on their assessment regarding the accuracy of a post: • Prior to you taking this study, were you aware of the existence of third-party fact-checking organizations (community-based fact-checking)?(Yes/No) • To what extent did the "Checked by third-party fact-checking organizations" tag ("Checked by other social media users with multiple perspectives tag", "Community note") influence you opinion about the accuracy of the social media posts?(5-point likert scale; Not at all, Slightly, Moderately, Very, Extremely) • We are interested in whether the "Checked by third-party fact-checking organizations" tag ("Checked by other social media users with multiple perspectives tag", "Community note") influenced your opinion about the accuracy of the social media posts that were tagged as potentially misleading.I rated "potentially misleading" posts as: (7-point likert scale; Much less accurate, Less accurate, Slightly less accurate, Tag had no influence, Slightly more accurate, More accurate, Much more accurate) • We are interested in whether the "Checked by third-party fact-checking organizations" tag ("Checked by other social media users with multiple perspectives tag", "Community note") influenced your opinion about the accuracy of the social media posts that were NOT tagged as potentially misleading.I rated posts that were NOT "potentially misleading" as: (7-point likert scale; Much less accurate, Less accurate, Slightly less accurate, Tag had no influence, Slightly more accurate, More accurate, Much more accurate) At the end of the survey, participants where asked to answer several demographic questions: age, gender, level of education, proficiency in English, US region where they live, stance toward god (or gods), whether they have been vaccinated against COVID-19, whether they see themselves as part of an ethnic minority, political orientation (Democrat, Republican, Third Party, Other), and questions on their voting behavior in the 2020 presidency election.First, they were asked who they voted for (Joe Biden, Donald Trump, Other Candidate, I did not vote for reasons outside my control, I did not vote but I could have, I did not vote out of protest) and second, who they would prefer to be president, if they absolutely had to choose between Joe Biden and Donald Trump.
In addition, participants had to provide an assessment of their attitudes toward the following statements on risk aversion and trust (5-point likert scale; Not at all, Not really, Undecided, Somewhat, Very much): • I am generally a person that is fully prepared to take risks.
• I usually have the feeling that I can trust the people I interact with in my daily life.
• I have a fundamental trust in democracy.
• I would rather do something that requires little thought than something that is sure to challenge my thinking abilities.

Fig. S2 .
Fig. S2.Difference in participants' responses to questions on demographics, beliefs, and CRT, depending on their political leaning.The values show the percentage share of pro-Republican/pro-Trump (red) and pro-Democrat/pro-Biden (blue) participants in the respective group.The column "Diff" reports the difference between both groups, "T" and "B" stand for Trump and Biden.

Fig. S3 .
Fig. S3.Community notes did not consistently reduce sharing intentions for misleading posts.Shown are the average marginal effects (AME) and 95 % confidence intervals from a hierarchical linear regression model with interaction terms predicting sharing intentions (7-point Likert scale normalized to the interval [0, 1]).AME is the difference in the average predicted sharing intentions between the group of interest and the reference group (e.g., an AME of +0.05 indicates a 5 percentage point difference in sharing intentions).(a) AMEs for misleading posts.(b) AMEs for non-misleading posts.(c) AMEs when replacing an expert flag with a community note (i.e., the average difference between the predicted marginal effects for community notes vs. expert flags) across the political leanings of participants (leaning Trump vs. Biden) and the political congruence of the fact-checked posts (concordant, neutral, discordant).(d) AMEs when replacing an expert flag with a community flag (i.e., the average difference between the predicted marginal effects for community flags vs. expert flags) across the political leanings of participants and the fact-checked posts.Control variables and random intercepts for posts and subjects were included.The 95 % confidence intervals (error bars)

Fig. S7 .
Fig. S7.Distribution of participants' responses regarding the political orientation of posts on 5-point Likert scales.
To assess representativeness beyond gender and political orientation, we looked at how other variables (demographics, beliefs, etc.) differed between participants preferring Trump vs. Biden.FigureS2visualizes the distributions of participants' responses for relevant variables.

Table S2 . Distribution of different exclusion criteria per condition.
TableS3reports the participants responses to questions regarding their social media behavior.79 % of all participants use Facebook, 64 % use Twitter/X, and 68 % use Instagram.Prior to all analyses, all participants not using social A.5. Social media behavior.

Table S6 . Average marginal effects (AME) and 95 % confidence intervals from a hierarchical linear regression model with four-way interaction terms predicting the perceived misleadingness of a post (7-point Likert scale normalized to the interval [0, 1]). AME is the difference in the average predicted misleadingness ratings between the group of interest and the reference group
(e.g., an AME of +0.05 indicates a 5

Table S7 . Estimation results for a hierarchical linear regression model with three-way interaction terms predicting the trustworthiness of a fact-check (7-point Likert scale normalized to the interval [0, 1]). Random intercepts for posts and subjects are included. N = 24,003 observations across 1,347 participants.
Significance: * * * p < 0.001; * * p < 0.01; * p < 0.05

Table S14 . Average marginal effects (AME) of replacing an expert flag with a community note depending on participants' reliance on fact-checks. The AME are calculated from hierarchical linear regression models with interaction terms predicting trust in fact-checks and misleadingness for misleading posts. Random intercepts for posts and subjects are included. The 95 % confidence intervals for the AMEs were derived using the bootstrap method for 1,000 resamples
A common method to assess the level of a persons' reflective thinking is the so-called Cognitive Reflection Test (CRT).Participants in our study were asked to answer a 4-item CRT (for further details see Section D).

Table S16 . Average marginal effects (AME) of different fact-checking interventions depending on the outcome of the CRT (= 1 if passed; = 0 otherwise) for misleading posts. The AME are calculated from linear mixed-effects regression models with interaction terms predicting trust in fact-checks and perceived misleadingness. Random intercepts for posts and subjects are included. The 95 % confidence intervals for the AMEs were derived using the bootstrap method for 1,000 resamples. N = 24,003 observations across 1,347 participants for DV
: T rustworthiness, and N = 64,454 observations across 1,810 participants for DV: M isleadingness.

Table S17 . Frequency of ratings on the level of response items (per post and per participant) across the experimental conditions. Ratings given by the participants on a 7-point Likert scale were rescaled into binary variables that took the value
= 1 if the rating was greater than Neutral (and = 0 otherwise).

Table S18 . Average marginal effects (AME) and 95 % confidence intervals from a hierarchical logistic regression model with three-way interaction terms predicting whether a fact-check was rated as trustworthy
(0 = no; 1 = yes).AME

Table S19 . Average marginal effects (AME) and 95 % confidence intervals from a hierarchical logistic regression model with four-way interaction terms predicting whether a post was rated as misleading
(0 = no; 1 = yes).AME