Trust & Safety (T&S) teams in the tech industry are currently at a “build or buy” crossroads. Waning budgets and mass layoffs, increased regulatory requirements, and new AI-based tools with impressive capabilities are changing T&S teams’ internal calculus on how to allocate limited resources to traditional T&S functions. T&S teams are now more frequently choosing between two paths: building and maintaining bespoke automated moderation systems or buying off-the-shelf services offered by third-party T&S vendors.
Third-party T&S vendors offer many services to their customers, which include social media companies, AI model developers, dating apps, online marketplaces, fundraising platforms, and more. Vendors offer to do it all: help online services develop content and conduct policies, enforce these policies at scale using automated tools, inform regulatory and legal compliance, and conduct risk assessments and audits, among other things. Increasingly, they are actively shaping norms and best practices for T&S across the field.
The vendorization of T&S, specifically the trend of companies outsourcing critical functions related to policy development and enforcement to third party private companies, can help online services when they don’t have internal capacity. At the same time, the reliance on third party guidance and tools can raise questions related to accountability, transparency, and users’ rights.
This rapid pace of change within the T&S field reflects a broader evolution of the T&S ecosystem and raises the stakes for accountability. T&S teams are often at the frontlines of determining who gets to use a platform, how users are expected to behave, and what they can say and do. Due to the outsized role many online services play in the information ecosystem, decisions made by T&S teams can have critical ramifications for users’ rights to access information and speak freely and securely. As a result, ensuring that T&S teams and the vendors they work with respect users’ freedom of expression specifically and human rights broadly is essential.
Some industry practitioners and academic researchers have written about the evolving landscape of T&S, focusing on recent changes, who vendors are, the services they provide, and through what processes to procure their services. How outsourcing key T&S functions to vendors affects internet users’ rights has been less discussed. This brief helps fill that gap by focusing on the human rights implications of outsourcing some T&S processes.
