One of the main hindrances to the success of automated fact-checking is the need for more fact checks and fact-checkers, Duke Reporters’ Lab director Bill Adair said at Wednesday’s IFCN Talk.
Adair, along with reporters lab lead technologist Christopher Guess and project manager Erica Ryan, offered their insights as part of a larger discussion of the Reporters’ Lab’s automated fact-checking project, Squash.
Adair said the goal was to create a program that could instantly surface on screen fact checks for scenarios such as broadcasts of political debates or live speeches. He showed how Squash takes claims from fact-checking partners submitted to ClaimReview — a fact check tagging system developed by the lab — and uses a matching algorithm to connect those fact checks to live speech. Squash also uses a program developed at the University of Texas called ClaimBuster to filter what the live speaker is saying into fact-checkable statements.
Adair acknowledged the system was not without its flaws, with Squash sometimes matching irrelevant fact checks or not tagging fact-checkable claims altogether. For that, the lab developed an interface called Gardener, which allows human intervention to ensure Squash is matching fact checks relevant to the claim being made.
But this system depends on having a wide variety of fact checks available to match the claims.
“We need more troops,” Adair said. While it’s relatively easy for fact-checkers to add their work to ClaimReview, Guess said the variety of topics being fact-checked is just as important as the total number of individual fact-checks.
“You need a corpus that can represent all the different topics somebody could be speaking about instead of just like 100 on capital gains tax,” he said. Adair argued that only comes with more funding and fact-checkers to do the work.
“It’s almost like Report for America, which is this army of journalists. What we need is like, fact check for the world,” Adair said. Ryan noted that Squash only pulled from the lab’s American fact-checking partners, which cut the number the team could use to train the algorithm, but Adair said there’s no reason this program could not be applied internationally.
One potential application could be to help fact-checkers who use WhatsApp tiplines. Fact-checkers use them to both collect claims and distribute their work to WhatsApp users (the platform’s peer to peer encrypted messaging features make it difficult for fact-checkers to track viral claims). Guess said matching algorithms could help streamline that process.
“It could turn it into more of an approve/disapprove instead of frantically searching your entire database by hand,” he said.
Spanish fact-checking organization Maldita.es has already had some success adding automation to its tipline, winning a 2021 European Press Prize for innovation in the process. Maldita.es used a combination of automated responses to fact-checking claims with the ability for human interventions to improve response times and quality for its audience.
Guess said automated fact-checking will always require human assistance. He noted that while an algorithm having 90% accuracy at matching fact checks to claims is considered excellent in the world of machine learning, the same doesn’t hold for fact-checking.
“That’s one in 10 claims being marked wrong, and when it comes to political misinformation even one in 10 being wrong fuels the flames even further,” Guess said. “What I always say is we build tools for fact-checkers not to replace fact-checkers.”
Adair said Squash would only come close to replacing fact-checkers if it passed what he called the “Aaron Sharockman challenge” — a reference to PolitiFact’s executive director. The challenge pits Squash against a team of live fact-checkers to see who can more quickly and accurately correct falsehoods during a political debate or speech.
“The question is who can hear what a politician says and find the relevant fact check fast in the database and retrieve it and display it,” he said. Some members of the audience suggested Adair’s team should test that theory at the upcoming virtual Global Fact 8 conference, taking place Oct. 20-23.
