Out of Tune: Fine-Tuning Foundation Models Leads to Unpredictable Safety Drift

In addition to CDT, this report was authored by Emaan Bilal and Dylan Hadfield-Menell of the Algorithmic Alignment Group at the Massachusetts Institute of Technology (MIT).

[ Read full report ]

General-purpose AI models are increasingly adapted by downstream developers for specialized uses — from medical transcription to legal contract analysis. This raises critical governance questions related to who should be held accountable for ensuring the safety of these systems as they are adapted throughout the supply chain. Some policy frameworks, including the EU AI Act, aim to address this by tying accountability to the degree to which models are modified, treating downstream developers as responsible only once a model has been “substantially” altered. But emerging research suggests this approach rests on a flawed assumption that how much a model is changed reliably predicts how safe those updates are.

In practice, fine-tuning — one prominent method for adapting general-purpose models — can affect model safety in unpredictable ways. Recent research we conducted in partnership with researchers at MIT’s Computer Science & Artificial Intelligence Laboratory underscores that even small changes to models can lead to substantial shifts in safety-relevant behavior, while more extensive changes can have little effect. In other words, seemingly minor and benign model changes can lead to “safety drift,” where a model’s behavior moves unexpectedly away from its behavior before modification.

Most research on how fine-tuning can weaken model safety has focused on cases where developers fine-tune models on malicious data to intentionally undermine safety guardrails, but even fine-tuning on seemingly harmless data can sometimes lead to less safe models. Indeed, as we find, safety characteristics can shift unpredictably after fine-tuning — in some cases degrading and in others improving. Safety drift doesn’t just affect model behavior on the tasks developers are tuning it to perform; it can also pop up in domains that are far removed from a model’s intended use. For example, fine-tuning a model to perform a specialized medical task risks making it more likely to produce advice on how to create and disseminate malware.

This research has notable implications for AI governance. Most importantly, it suggests that governance approaches anchored to how much a model has changed risk misallocating responsibility by overlooking how safety actually changes. Addressing this issue requires rethinking three things: (1) how responsibility is allocated across the AI supply chain, recognizing that upstream providers and downstream deployers bring different forms of expertise to ensuring safe use; (2) what shared technical infrastructure is needed to support safety at every stage of development and deployment, especially when models or the systems they are embedded in can be modified at different points in that chain; and (3) what transparency and information-sharing mechanisms are most effective for enabling coordination around emergent risks.

Key Takeaways

• When models are fine-tuned, they can experience “safety drift” — that is, their safety behavior can become stronger or weaker in unexpected ways, both for their intended application as well as in other contexts.
• These shifts can happen even if the goal of modification is benign and the intended use case is run-of-the-mill.
• The amount of model modification is not a reliable signal of safety impact. Policy or governance frameworks that rely on such proxies to trigger testing of safety obligations risk under-testing or misplacing accountability when safety failures occur.
• Safety characteristics that are at risk of shifting after modification can be general or context specific; upstream providers understand and are often more capable of assessing general safety risks, while downstream deployers are more likely to understand scoped, domain-specific ones. Responsibility for AI safety should follow at least to some degree the expertise and resourcing of actors in the supply chain.
• Reliable and valid evaluations — and tools to facilitate it — are sorely needed, including domain-specific safety assessments that reflect real-world deployment and support deployers in reassessing general safety after fine-tuning.
• Effective governance of AI in general, and addressing safety drift in particular, requires new approaches to transparency and coordination between upstream and downstream developers about safety risks introduced through model modification. Government agencies and third-party actors (like academic researchers, AI safety institutes, independent evaluators, and multi-stakeholder coalitions) play a complementary role by advancing domain-specific evaluation research, maintaining open and independent testing infrastructure, and centralizing and disseminating findings.

The Challenge of Model Modification for AI Supply Chain Governance

AI models that power applications like ChatGPT and Claude are often referred to as “foundation models” because they serve as a base for a wide range of downstream applications. These models can be adapted for tasks as varied as medical transcription or identifying crop diseases from images. This flexibility is a major source of their power, but also contributes to governance and policy challenges.

When downstream developers modify foundation models to perform specific tasks, they can also meaningfully change the risks those systems pose. Yet most policy attention has focused on regulating foundation models themselves. The EU AI Act, for example, includes specific provisions for so-called general-purpose AI (GPAI) systems, including transparency requirements, and for models that may pose a certain level of risk, more stringent requirements related to evaluation, risk management, and reporting.

However, if both functionality and risk evolve as models move through the AI supply chain, this raises difficult questions about what kinds of transparency are actually useful at different stages, which measurements and safeguards are appropriate, and who should be accountable when systems fail.

Recently, researchers have begun to examine how modifying AI models can affect their safety behavior. Much of this work has focused on fine-tuning, a common adaptation technique in which developers retrain general-purpose models on relatively small datasets to specialize them for particular tasks. These studies show that even minimal fine-tuning can have outsized effects including in areas well beyond the domain for which a model is fine-tuned. For example, models fine-tuned to generate insecure code have been shown to exhibit broader forms of misalignment, including producing responses that encouraged violent wrongdoing, such as advising a user to murder a spouse in a moment of frustration or encouraging a bored user to abuse drugs. Fine-tuning can also undermine model safety even when the fine-tuning dataset contains no explicitly harmful data. This sort of research has indicated that fine-tuning even with small or widely-used datasets can meaningfully affect model safety, but it remains unclear how much modification is required before a model’s safety behavior substantively changes, and whether certain technical choices around how to fine-tune models tend to amplify or mitigate such shifts.

This uncertainty poses a challenge for regulatory approaches that take a supply-chain view of AI risk, where responsibility for failures depends on how different actors modify and deploy models. For example, the EU AI Act’s implementation guidance assigns different obligations to downstream modifiers based on how extensively they adapt a model even though the relationship between the degree of modification and resulting safety impacts remains poorly understood. Without stronger empirical evidence on real-world fine-tuning practices, policymakers risk relying on crude technical proxies to allocate responsibility that could overlook issues presented by adapted models deployed in high-stakes settings.

The unpredictable effects of model modifications on safety behavior in high-stakes domains

In our research, we set out to better understand how deployers’ fine-tuning practices affect model safety in real-world, high-stakes domains. As a first step, we compared publicly available models fine-tuned for medical and legal tasks with their corresponding base models, conducting general safety evaluations and domain-specific assessments tailored to medicine and law on both the base and fine-tuned models.

Across these real-world models, we observed substantial variation in safety outcomes. Some fine-tuned models appeared safer than their base counterparts, while others were markedly less safe. Crucially, these differences depended heavily on how safety was measured: most fine-tuned models performed better on at least one safety evaluation while performing worse on others. These findings suggest that the impacts of fine-tuning on safety vary, but do not explain what might drive those shifts. To explore whether specific fine-tuning choices could explain this variability, we also ran controlled experiments in which we fine-tuned base models on medical and legal data, varying some aspects of the setup and holding others constant.

We found that no single fine-tuning choice or combination of choices consistently explained why models became safer in some cases and less safe in others, either for domain-specific or general safety evaluations. The extent of model change also failed to offer a clear explanation. Even relatively small adaptations sometimes produced large shifts in safety behavior, and more substantial tuning did not reliably lead to greater safety degradation. Across both medical and legal models, safety outcomes remained highly variable and difficult to predict.

Together, our findings show that fine-tuning can change a model’s safety in unpredictable ways, making it difficult to know in advance whether a modified system will be safer or riskier. As such, developers and policymakers should not assume that fine-tuned models will inherit the safety properties of their original versions. As a result, adapted models should be treated as distinct systems and evaluated on their own, especially in high-stakes settings.

Revisiting AI Governance and Policy for Adapted Models

Our results, which converge with existing research on the safety impacts of fine-tuning, have several important implications for how governance professionals and policymakers should think about managing risk along the AI supply chain. In particular, the fact that model modification has the potential to shift both the general safety profile of adapted models as well as safety considerations within their intended domain of application complicates questions about how to allocate responsibility among relevant actors.

Assigning Accountability Throughout the Supply Chain

To a first approximation, it may seem that our results would suggest that downstream developers should be held responsible for any safety failures of the models they deploy; after all, if their modifications are what causes a shift in safety from the base model, then perhaps they should be responsible for ensuring the safety of models after deployment?

However, a simple analogy helps clarify the limits of placing responsibility solely on downstream deployers. Imagine an engine manufacturer that sells engines with elements that can be adjusted for different driving conditions. If commonplace, well-intentioned adjustments routinely led to unpredictable safety failures, it would probably not be reasonable to place responsibility only on the mechanic doing the tuning. The manufacturer likely would also be expected to design engines that are safely adjustable and/or to provide clear guidance on tuning procedures that preserve assumptions about the engine’s safety.

Viewed this way, foundation models might resemble the engines and fine-tuning is the process of adjusting them to work best in specific driving conditions. If safety failures commonly emerge from ordinary tuning practices, meaningful governance will likely need to extend beyond downstream deployers to include upstream model providers. Indeed, we posit that safety drift is an emergent property of the entire AI value chain, shaped by interactions among base models, tuning data and methods, and downstream design choices. Effective AI governance therefore needs to reflect the structure of the AI supply chain, accounting for large asymmetries in both resources and expertise between foundation model providers and smaller deployers.

Our findings also have clear implications for how model modification should be treated in terms of assigning accountability in policy and governance frameworks, such as the EU AI Act. Because fine-tuning can alter a model’s safety profile in unpredictable ways, rigid technical thresholds risk missing precisely the changes that matter most. Our findings suggest that relatively small, seemingly benign modifications can produce substantial safety shifts, while more extensive adaptations can sometimes improve safety. What matters is not how large or intensive a modification is, but how it affects safety-relevant behavior. A governance approach centered on observed safety impact, rather than the size of the modification, would better align accountability along the supply chain with realized risk.

Improving the Ecosystem for AI Evaluation

Governing AI systems based on the risks they pose in the real world ultimately requires better ways of measuring those risks. While many general safety evaluations are available today, it remains unclear how well they predict real-world safety outcomes. Improving understanding of how these sorts of pre-deployment risk measurements relate to post-deployment harms will be essential to develop safety impact-informed thresholds.

Our research also highlights a gap in domain-specific safety measurement. Existing general-purpose safety assessments may not remain valid once a model has been fine-tuned for a specific domain. For example, benchmarks designed around single-turn prompts and generic response styles may fail to capture risks that emerge in the multi-step, task-specific interactions that fine-tuned models are optimized for. Addressing this gap will require institutional investment in domain-specific safety methods that are designed with post-modification behavior in mind, ideally through open-source efforts that make these resources broadly accessible. Making evaluation infrastructure accessible to resource-constrained downstream developers, who often lack the compute and expertise to conduct comprehensive safety testing, will also be essential to translating evaluation research into practice.

Enabling Coordination and Communication through AI Transparency

Our findings also point to the need for new approaches to information sharing and transparency that reflect how risks can evolve — sometimes substantially — across the AI supply chain. Traditionally, researchers and responsible AI advocates have emphasized static documentation tools, such as model and system cards, to communicate intended uses, known risks, and key aspects of a system’s development. These tools can work especially well for relatively stable, traditional machine-learning systems with narrowly bounded outputs, such as image classifiers or price-prediction models.

Generative systems present a different challenge. Foundation models and their fine-tuned derivatives can produce a wide range of outputs and be adapted for many different use cases, highlighting the limitations of static documentation. In the absence of a clearly defined deployment context, practitioners often face uncertainty about which evaluations to run, which risks to document, and how responsibility for documentation should be divided across the supply chain.

To better manage the risks introduced by model modification, developers and deployers need more flexible ways to share information across the AI supply chain. Foundation model providers can use documentation to describe known limitations in how well their models’ safety properties hold up under fine-tuning, as well as the evaluations they have conducted and their constraints. But static, one-directional documentation alone is not sufficient. Downstream developers also need channels to communicate back to upstream providers when they observe new or unexpected risks, both within specific application contexts and beyond them. This feedback can help foundation model developers identify gaps in their safety practices, strengthen their safeguards, and update their documentation to support others facing similar challenges. In this way, documentation can become a living resource that responds to real-world use and supports more responsible deployment.

Conclusion

Our research reinforces that AI safety remains fragile, and even common, benign modifications can introduce unpredictable risks. Notably, the finding that significant safety drift can occur regardless of the relative magnitude of a modification challenges governance frameworks that use compute thresholds or modification magnitude to trigger oversight. Foundation model providers, downstream developers, government agencies, and independent researchers each have distinct and complementary roles to play in building a more resilient ecosystem for safe AI development and deployment downstream.

Read the full report.

Read the academic paper.