Outside Looking In: Approaches to Content Moderation in End-to-End Encrypted Systems

A new front has opened up in the Crypto Wars: content moderation. During the 1990’s, policy debates in the U.S. and Europe about encryption focused on the benefits and risks of public and foreign access to encryption. Law enforcement and intelligence agencies around the world pushed for restrictions on the development and export of encryption technologies, arguing that greater public access would limit their ability to monitor communications to fight crime and protect the public. In the end, the U.S. government decided against such restrictions with a shift in policy in 1999 (Swire & Ahmad, 2011), and other governments followed suit.

As billions of people around the world began to use encrypted services to protect their privacy and data when communicating with others, the concerns of law enforcement agencies regained prominence in the last decade. In 2014, the then-Director of the FBI argued that encrypted communications were an impediment to law enforcement (Federal Bureau of Investigation, 2014). A 2020 statement by the governments of the U.S., UK, Canada, India, Japan, Australia, and New Zealand expressed similar concerns, calling for greater access by law enforcement to encrypted communications (U.S. Department of Justice, 2020).

Statements such as these tend to focus encryption policy on law enforcement and intelligence agencies’ claims that they need to be able to access encrypted communications (National Academies of Sciences, Engineering, and Medicine, 2018). But encryption is not just a law enforcement issue. The availability of secure encrypted communication services is central to privacy, free expression, and the security of today’s online commerce (Thompson & Park, 2020).

Perhaps recognizing the uphill battle they face to undermine such a crucial part of our online infrastructure, some law enforcement officials have begun to link the threat of unconstrained illegal content online to concerns about large social media platforms’ content moderation practices. In the U.S., for example, the proposed EARN IT Act was framed as a bill that would establish best practices in content moderation for fighting child sexual abuse material (CSAM), but the debate quickly came to focus on the implications of the bill for end-to-end encryption (E2EE), with many commentators expressing concern that the bill’s approach was designed to discourage providers from offering E2EE services or create strong incentives to build in a special access mechanism for law enforcement (Murdock, 2020; Newman, 2020; Ruane, 2020).

But what is the actual effect of encryption on content moderation?

In this paper, we assess existing technical proposals for content moderation in E2EE services. First, we explain the various tools in the content moderation toolbox, how they are used, and the different phases of the moderation cycle, including detection of unwanted content. We then lay out a definition of encryption and E2EE, which includes privacy and security guarantees for end-users, before assessing current technical proposals for the detection of unwanted content in E2EE services against those guarantees.

We find that technical approaches for user-reporting and meta-data analysis are the most likely to preserve privacy and security guarantees for end-users. Both provide effective tools that can detect significant amounts of different types of problematic content on E2EE services, including abusive and harassing messages, spam, mis- and disinformation, and CSAM, although more research is required to improve these tools and better measure their effectiveness. Conversely, we find that other techniques that purport to facilitate content detection in E2EE systems have the effect of undermining key security guarantees of E2EE systems.

The current technical proposals that we reviewed all focus on content detection, which is only one part of the content moderation process. Thus, there may be other useful and effective approaches to moderation for countering abuse in E2EE systems, including user education about applicable policies, improved design to encourage user reports, and consistency of enforcement decisions. These approaches may offer important potential avenues for researchers to build on our analysis.

Read the full report here. (Updated Jan. 13, 2022)

This CDT report was also translated into Portuguese by the Institute for Research on Internet and Society (IRIS).

---