Can Community Notes Replace Professional Fact-Checkers?

🔼 This figure shows a breakdown of the types of sources cited by authors of Community Notes on Twitter/X. It indicates the percentage of links used that fall into various categories, such as News, Social Media, Fact-Checking Organizations, Government sites, Academic sources, and others. This helps to illustrate the breadth and nature of the evidence cited in support of community notes’ claims.

read the caption

Figure 2: The categories of links used by Community notes’ authors as a source.

🔼 This bar chart displays the mean scores of community annotations for misleading posts categorized by type. Each bar represents a category of misleading posts, such as ‘Misleading Factual Error’ or ‘Misleading Manipulated Media,’ and its height shows the average score assigned by community annotators. The scores likely reflect the degree to which the community considered the posts to be misleading, with higher scores indicating a greater perceived level of misleadingness.

read the caption

Figure 3: Mean scores of community annotations of misleading posts.

🔼 Figure 4 presents two subfigures illustrating how community notes engage with fact-checking resources in the context of broader misinformation narratives. Subfigure (a) categorizes the strategies used in community notes to debunk claims related to these narratives, while subfigure (b) shows how fact-checking sources are specifically employed within these debunking strategies. This visual representation helps to understand the interplay between community-driven fact-checking and professional fact-checking sources in combating misinformation.

read the caption

Figure 4: (a) strategies in debunking claims related to broader narratives. (b) the different ways in which fact-checking sources are used to debunk claims.

🔼 This figure shows a bar chart comparing the distribution of topics across community notes that do and do not include fact-checking sources. It illustrates which topics are more or less likely to be addressed by notes that cite fact-checkers, offering insights into the relationship between fact-checking and community moderation.

read the caption

Figure 5: Distribution of notes’ topics, with and without a fact-checking source.

🔼 This figure shows the number of community notes created each month, categorized by their helpfulness rating (helpful, not helpful, or needs more data). The x-axis represents time (months), and the y-axis represents the count of notes. The grey vertical line highlights December 2022, marking when the community notes feature became globally accessible. This visualization provides insight into the growth and user engagement with the community notes system over time.

read the caption

Figure 6: A histogram of the number of community notes written every month and their rating (helpful, not helpful, or needs more data. The grey vertical line (December 2022) indicates the date when the community notes became visible worldwide.

🔼 This figure shows the distribution of link categories used by authors of community notes that received a ‘helpful’ rating. It provides a breakdown of the types of sources cited to support claims within these notes. Categories include news websites, social media, academic sources, government sources, etc. The percentages illustrate the relative frequency of each source type within the ‘helpful’ community notes.

read the caption

Figure 7: The categories of links used by Community notes’ authors as a source, filtering for notes rated as “helpful”.

🔼 This figure shows the distribution of link categories used in community notes that received a ’not helpful’ rating. It breaks down the types of sources cited by users in their notes, such as news websites, social media, academic sources, fact-checking organizations, and others. This provides insight into the types of sources deemed unhelpful by the community in their efforts to provide context to posts.

read the caption

Figure 8: The categories of links used by Community notes’ authors as a source, filtering for notes rated as “not helpful”.

🔼 This figure displays a bar chart showing the average community ratings for notes categorized by whether or not they include a fact-checking source. It visually compares the mean rating scores of notes that cite fact-checking sources to those that do not, allowing for a direct comparison of perceived helpfulness and quality based on the presence or absence of such references.

read the caption

Figure 9: Community ratings of notes with and without fact-checking source.

🔼 This figure details the annotation scheme used in the study to analyze community notes on Twitter. It shows a table outlining the twelve binary attributes used to characterize each (tweet, note) pair. These attributes capture various aspects of the interaction, such as whether the note addresses a broader narrative or conspiracy theory, discredits the source, adds missing context, highlights AI-generated content or edited media, and includes links to different types of sources (e.g., direct source, official source, scientific source, fact-checking source, etc.). The table also specifies if fact-checking is performed within the note or through external links. Each attribute can be checked (☑) or unchecked (☐) to reflect the presence or absence of that feature in the tweet and its associated community note.

read the caption

Figure 10: Our annotation setup.

🔼 This table presents the relationship between the presence of fact-checking sources and the topic of community notes. It shows the percentage of notes discussing a broader narrative or conspiracy theory that also include a fact-checking source, compared to the percentage of notes on other topics that include a fact-checking source. This helps to illustrate the importance of fact-checking in addressing complex or controversial claims.

read the caption

Table 2: Percentage of samples related to a broader narrative or conspiracy vs. have a fact-checking source.

🔼 This table lists fact-checking organizations that are signatories to the International Fact-Checking Network (IFCN). For each organization, the table provides its name, website URL, the languages its content is available in, and its primary region or area of focus. This information is relevant to the study’s analysis of how community notes on social media platforms utilize and reference professional fact-checking sources.

read the caption

Table 3: List of professional fact-checking organisations and their URLs.

🔼 This table lists the top 100 most frequently appearing website domains in the Twitter/X Community Notes dataset. For each domain, the table provides a corresponding category. The categories used for classification reflect the general nature of the websites, such as social media, news, government, academic institutions, blog posts, fact-checking organizations, databases, commercial entities, reference sites, non-profit organizations, and an ‘other’ category for miscellaneous websites. The categorization helps understand the types of sources cited within Community Notes to support or refute claims.

read the caption

Table 4: List of top 100 most common domains found in the community notes dataset, and their categorization.

🔼 This table presents eight examples of community notes that cite fact-checking sources but were rated as unhelpful due to missing or unreliable information. Each row shows an ID, a summary of the claim addressed, and links to the fact-checking sources cited. The ’not helpful’ rating suggests that while the notes attempted to debunk misinformation using external resources, the provided information or the way it was presented was ultimately insufficient or flawed.

read the caption

Table 5: Examples of community notes containing fact-checking sources that are rated as having notHelpfulSourcesMissingOrUnreliable.