Content moderation rules used to be a question of taste. Now, they can determine a service’s prospects for survival.
6 min
Every public communication platform you can name—from Facebook, Twitter and YouTube to Parler, Pinterest and Discord—is wrestling with the same two questions:
How do we make sure we’re not facilitating misinformation, violence, fraud or hate speech?
At the same time, how do we ensure we’re not censoring users?
The more they moderate content, the more criticism they experience from those who think they’re over-moderating. At the same time, any statement on a fresh round of moderation provokes some to point out objectionable content that remains. Like any question of editorial or legal judgment, the results are guaranteed to displease someone, somewhere—including Congress, which this week called the chief executives of Facebook, Google and Twitter to a hearing on March 25 to discuss misinformation on their platforms.
For many services, this has gone beyond a matter of user experience, or growth rates, or even ad revenue. It’s become an existential crisis. While dialling up moderation won’t solve all of a platform’s problems, a look at the current winners and losers suggests that not moderating enough is a recipe for extinction.
Facebook is currently wrestling with whether it will continue its ban of former president Donald Trump. Pew Research says 78% of Republicans opposed the ban, which has contributed to the view of many in Congress that Facebook’s censorship of conservative speech justifies breaking up the company—something a decade of privacy scandals couldn’t do.
Parler, a haven for right-wing users who feel alienated by mainstream social media, was taken down by its cloud service provider, Amazon Web Services, after some of its users live-streamed the riot at the U.S. Capitol on Jan. 6. Amazon cited Parler’s apparent inability to police content that incites violence. While Parler is back online with a new service provider, it’s unclear if it has the infrastructure to serve a large audience.
During the weeks Parler was offline, the company implemented algorithmic filtering for a few content types, including threats and incitement, says a company spokesman. The company also has an automatic filter for “trolling” that detects such content, but it’s up to users whether to turn it on or not. In addition, those who choose to troll on Parler are not penalized in Parler’s algorithms for doing so, “in the spirit of First Amendment,” says the company’s guidelines for enforcement of its content moderation policies. Parler recently fired its CEO, who said he experienced resistance to his vision for the service, including how it should be moderated.
Now, just about every site that hosts user-generated content is carefully weighing the costs and benefits of updating their content moderation systems, using a mix of human professionals, algorithms and users. Some are even building rules into their services to pre-empt the need for increasingly costly moderation.
The saga of gaming-focused messaging app Discord is instructive: In 2018, the service, which is aimed at children and young adults, was one of those used to plan the Charlottesville riots. A year later, the site was still taking what appeared to be a deliberately laissez-faire approach to content moderation.
By this January, however, spurred by reports of hate speech and lurking child predators, Discord had done a complete 180. It now has a team of machine-learning engineers building systems to scan the service for unacceptable uses, and has assigned 15% of its overall staff to trust and safety issues.
This newfound attention to content moderation helped keep Discord away from the controversy surrounding the Capitol riot, and caused it to briefly ban a chat group associated with WallStreetBets during the GameStop stock runup. Discord’s valuation doubled to $7 billion over roughly the same period, a validation that investors have confidence in its moderation strategy.
The challenge successful platforms face is moderating content “at scale,” across millions or billions of pieces of shared content.
Before any action can be taken, services must decide what should be taken down, an often slow and deliberative process.
Imagine, for example, that a grass-roots movement gains momentum in a country, and begins espousing extreme and potentially dangerous ideas on social media. While some language might be caught by algorithms immediately, a decision about whether discussion of a particular movement, like QAnon, should be banned completely, could take months on a service such as YouTube, says a Google spokesman.
One reason it can take so long is the global nature of these platforms. Google’s policy team might consult with experts in order to consider regional sensitivities before making a decision. After a policy decision is made, the platform has to train AI and write rules for human moderators to enforce it—then make sure both are carrying out the policies as intended, he adds.
While AI systems can be trained to catch individual pieces of problematic content, they’re often blind to the broader meaning of a body of posts, says Tracy Chou, founder of content-moderation startup Block Party and former tech lead at Pinterest.
Take the case of the “Stop the Steal” protest, which led to the deadly attack on the U.S. Capitol. Individual messages used to plan the attack, like “Let’s meet at location X,” would probably look innocent to a machine-learning system, says Ms Chou, but “the context is what’s key.” Facebook banned all content mentioning “Stop the Steal” after the riot.
Even after Facebook has identified a particular type of content as harmful, why does it seem constitutionally unable to keep it off its platform?
It’s the “prevalence problem.” On a truly gigantic service, even if only a tiny fraction of content is problematic, it can still reach millions of people. Facebook has started publishing a quarterly report on its community standards enforcement. During the last quarter of 2020, Facebook says users saw seven or eight pieces of hate speech out of every 10,000 views of content. That’s down from 10 or 11 pieces the previous quarter. The company said it will begin allowing third-party audits of these claims this year.
While Facebook has been leaning heavily on AI to moderate content, especially during the pandemic, it currently has about 15,000 human moderators. And since every new moderator comes with a fixed additional cost, the company has been seeking more efficient ways for its AI and existing humans to work together.
In the past, human moderators reviewed content flagged by machine learning algorithms in more or less chronological order. Content is now sorted by a number of factors, including how quickly it’s spreading on the site, says a Facebook spokesman. If the goal is to reduce the number of times people see harmful content, the most viral stuff should be top priority.
Companies that aren’t Facebook or Google often lack the resources to field their own teams of moderators and machine-learning engineers. They have to consider what’s within their budget, which includes outsourcing the technical parts of content moderation to companies such as San Francisco-based startup Spectrum Labs.
Through its cloud-based service, Spectrum Labs shares insights it gathers from any one of its clients with all of them—which include Pinterest and Riot Games, maker of League of Legends—in order to filter everything from bad words and human trafficking to hate speech and harassment, says CEO Justin Davis.
Mr Davis says Spectrum Labs doesn’t say what clients should and shouldn’t ban. Beyond illegal content, every company decides for itself what it deems acceptable, he adds.
Pinterest, for example, has a mission rooted in “inspiration,” and this helps it take a clear stance in prohibiting harmful or objectionable content that violates its policies and doesn’t fit its mission, says a company spokeswoman.
Services are also attempting to reduce the content-moderation load by reducing the incentives or opportunity for bad behaviour. Pinterest, for example, has from its earliest days minimized the size and significance of comments, says Ms Chou, the former Pinterest engineer, in part by putting them in a smaller typeface and making them harder to find. This made comments less appealing to trolls and spammers, she adds.
The dating app Bumble only allows women to reach out to men. Flipping the script of a typical dating app has arguably made Bumble more welcoming for women, says Mr Davis, of Spectrum Labs. Bumble has other features designed to pre-emptively reduce or eliminate harassment, says Chief Product Officer Miles Norris, including a “super block” feature that builds a comprehensive digital dossier on banned users. This means that if, for example, banned users attempt to create a new account with a fresh email address, they can be detected and blocked based on other identifying features.
Facebook CEO Mark Zuckerberg recently described Facebook as something between a newspaper and a telecommunications company. For it to continue being a global town square, it doesn’t have the luxury of narrowly defining the kinds of content and interactions it will allow. For its toughest content moderation decisions, it has created a higher power—a financially independent “oversight board” that includes a retired U.S. federal judge, a former prime minister of Denmark and a Nobel Peace Prize laureate.
In its first decision, the board overturned four of the five bans Facebook brought before it.
Facebook has said that it intends the decisions made by its “supreme court of content” to become part of how it makes everyday decisions about what to allow on the site. That is, even though the board will make only a handful of decisions a year, these rulings will also apply when the same content is shared in a similar way. Even with that mechanism in place, it’s hard to imagine the board can get to more than a tiny fraction of the types of situations content moderators and their AI assistants must decide every day.
But the oversight board might accomplish the goal of shifting the blame for Facebook’s most momentous moderation decisions. For example, if the board rules to reinstate the account of former President Trump, Facebook could deflect criticism of the decision by noting it was made independent of its own company politics.
Meanwhile, Parler is back up, but it’s still banned from the Apple and Google app stores. Without those essential routes to users—and without web services as reliable as its former provider, Amazon—it seems unlikely that Parler can grow anywhere close to the rate it otherwise might have. It’s not clear yet whether Parler’s new content filtering algorithms will satisfy Google and Apple. How the company balances its enhanced moderation with its stated mission of being a “viewpoint neutral” service will determine whether it grows to be a viable alternative to Twitter and Facebook or remains a shadow of what it could be with such moderation.
Copyright 2020, Dow Jones & Company, Inc. All Rights Reserved Worldwide. LEARN MORE
The Australian leather house has opened an immersive four-day pop-up in Manhattan, unveiling its Bloom Collection and redefining what a product launch can look like.
Following the successful launch of its Palais Collection, MAISON de SABRÉ has unveiled a new modular handbag system offering more than 720 styling combinations.
Randy Boyagoda’s new novel, “Lords of Serendipity,” delivers a biting satire of cheating, politics and dysfunction across modern universities.
5 min
Administrative cowardice. Rampant cheating. The soaking and scamming of undergrads and grad students. Performative wokeism. Intensifying competition for international tuition dollars from an expansive China.
These are either the latest dire headlines about American higher ed or the myriad subplots of “Lords of Serendipity,” an ambitious new novel by Randy Boyagoda. Unlike most campus fiction, its scope extends well beyond the cloister of the liberal arts classroom to whammy the entire university system, from its financial model to its function within a globalized economy. Along the way, it skewers its lightning-rod targets with a precision too sharp to be considered entirely fictional.
That its author is not only well-versed in satire, having written four other novels in the genre, but is also a professor of English at the University of Toronto and an experienced college administrator, further blurs the lines between comic fiction and brutal fact. It’s the kind of no-holds-barred book only a consummate insider would know to write—but only one who is also conveniently outside the American university system could get away with writing.
“This is someone who’s got the inside dope on what can go on at universities, especially with international students, and what kind of con games are going on,” said Margaret Atwood, a friend and fellow Torontonian.
The novel, like Boyagoda himself, is often very funny—even as the dilemmas its characters face read as tragedy. Its cast includes the bickering members of a Sri Lankan hotel cleaner’s family, insecure American freshmen, dead-end third-tier college professors and their highflying Ivy League counterparts and dispirited university functionaries. An old-fashioned Dickensian tale, it sweeps across continents and campuses, from a classic liberal-arts college to a stand-in for Harvard University to an urban technical school to the dilapidated buildings of a state-run university in Sri Lanka.
“I’m a joyful person,” Boyagoda said in an interview, though he prefers to see his outlook on academia as “hopeful” rather than optimistic. He smiles and laughs frequently. He bikes everywhere. He derives energy from his students and from his four daughters, ages 14 to 20. They help him and his American wife, Anna, also an educator, care for Boyagoda’s father, an immigrant from Sri Lanka who has dementia and lives with the family.
The germ of the novel came to Boyagoda during the pandemic, when he was serving as a vice-dean. An urban legend was circulating about a statistics grad student selling passing grades to international students. The cheating scheme itself may well have been apocryphal, but it felt tantalizingly plausible.
“As a novelist, it stayed with me,” Boyagoda said. “Who would sell grades for a stats course? Why would someone buy grades? What’s motivating them? Why wouldn’t the university be doing something about it?” What he didn’t want to do, he explained, was write “another novel about a creative writing professor and his drama with students.”
In addition to writing novels—the first of which was a finalist for Canada’s prestigious Giller Prize; he’s written eight books in total—Boyagoda teaches literature full-time. For his students next semester, kids raised in a purportedly “post-literature era,” he’s assigning three 700-page novels, he noted mischievously, books by George Eliot, Ralph Ellison and Kiran Desai.
On RateMyProfessors, students remark on Boyagoda’s heavy assignments, tough grading and distaste for cellphones, but also hail his “comedy.” When alerted to these reviews, Boyagoda laughed. “Guilty as charged,” he said.
“Randy’s one of those people in academia who is very well educated but doesn’t have an ounce of superiority about him,” said John Irving, who frequently guest lectures in Boyagoda’s classroom. “It’s amazing to see how willing students are to talk in his classrooms, how little afraid they feel of being mocked or put down—he gets complete candor out of them.”
Negotiating competing interests seems to come naturally to him. Like many college campuses, after the Oct. 7 Hamas attack on Israel, the University of Toronto was embroiled in protests, encampments and discord; part of the school’s response was to convene a working group on civil discourse. Boyagoda was put in charge, serving for two years. It’s the kind of thankless task many professors would flee from, but Boyagoda relished the prospect and thinks the group, which is ongoing, made progress.
He has since pivoted to another formidable academic concern: the waning prestige of the humanities. In July, Boyagoda became the director of the Jackman Humanities Institute and special advisor on the humanities to the dean of arts and science. This fall, he is starting a series of gatherings he calls the Dead Book Club. (Dead writers feature heavily among his favorites: St. Augustine, Dante, William Faulkner, Evelyn Waugh, Saul Bellow.)
“The goal is to encourage students to read for its own sake and have conversation for its own sake,” Boyagoda said. “The importance of the humanities will be revealed to those who participate.”
“Lords of Serendipity,” which publishes next week, reflects Boyagoda’s commitment to the pursuit of knowledge: “I’m supposed to say I learn more from my students than they do from me, but no,” Boyagoda said. “I enjoy sharing my excitement and joy about literature and ideas. There are these moments when you are teaching and you can see someone’s face change because they didn’t know something and then they did. That moment when you’re actually watch someone enlarge themselves is deeply satisfying.”
In literary circles, Boyagoda is often referred to as a Catholic novelist, a description he embraces. “A novelist willing to give shape and voice to contemporary religious experience is needed,” Boyagoda said. “I understand myself as a Catholic novelist, provided that I’m not only a Catholic novelist, I’m also a South Asian novelist and I’m a Canadian novelist. I don’t want to be captive to one label.”
His characters, he said, are subjected to what he calls “Graham Greene character situations.”
“They are trying to do good in a fallen world and realize the most they can do is less harm than someone else would do in that same role,” he explained. “They have to accept they are complicit in something that provides some good thing to others. It’s a very Graham Greene way of thinking about sacrifice.”
The campus green they inhabit is a febrile zone in which lofty ideals and cynicism regularly collide. It’s a place where a traditional holiday college celebration is now an occasion for political protest. Kids freely police one another inside and out of the classroom with social media their constant, ready weapon of choice. International students, knowing their family’s financial futures hinge on their degrees, operate in terror of failure and deportation.
Early readers of the book include Gary Shteyngart and Junot Diaz, both of whom have taught at universities and written comic novels. They each contacted Boyagoda midway through to say, in essence, “You nailed it.”
As biting as “Lords of Serendipity” can be, Boyagoda’s faith in higher education remains undaunted.
“It’s absurd that we live in societies that make the decision to take people between the ages of 18 and 22, when they are most physically capable of contributing to our nations and communities, and tell them, ‘Go away for four years, and read and learn and have conversations,’” he said. “I want students to be aware of what an incredible privilege that is.”
Corrections & Amplifications
Randy Boyagoda is a special advisor on the humanities to the dean of arts and science at the University of Toronto. An earlier version of this article incorrectly said he is a special adviser to the dean of humanities. (Corrected on Sept. 2)
