How Foundation Model Safety Training Works

Introduction

Foundation models — generative AI models, like those used in OpenAI’s ChatGPT, that can perform a wide variety of tasks — are remarkably capable, but their capabilities come with significant risks. Without proper safeguards, foundation models could, for instance, generate harmful content like nonconsensual deepfakes, provide dangerous advice (for instance, unacceptably risky medical advice), or assist with malicious activities like cyberattacks or influence campaigns.

Foundation model developers use several techniques to mitigate these risks. One important type of risk mitigation is “safety training,” or practices that aim to make foundation models behave in ways their developers intend them to, and ensure models respond in useful, non-harmful ways to users’ queries. This explainer explores the technical methods developers use to shape AI behavior, the people and processes involved in creating safety training data, and the principles that guide the safety training process.

At the same time, this explainer does not cover everything developers do to prevent their models from causing harm: for instance, it does not include the system-level content filters and automated monitoring systems that developers employ. While system-level safeguards are undoubtedly important and may be an important focus for transparency as well, we avoid discussing them here so that this report is not too complex and unwieldy.

Read this explainer.

Read the full report, entitled “Tuning Into Safety: How Visibility into Safety Training Can Help External Actors Mitigate AI Misbehavior.”