Beyond High-Risk Scenarios: Recentering the Everyday Risks of AI

This blog was authored by Stephen Yang, CDT Summer Fellow for the CDT AI Governance Lab.

Developers and regulators have spent significant energy addressing the highly consequential –– often life-altering –– risks of artificial intelligence (AI), such as how a misdiagnosis of an AI-powered clinical decision support system could be a matter of life and death.

But what about the more prosaic scenarios where AI causes harm? An error in an AI transcription system may lead to complications in insurance reimbursement. An average person’s likeness may be co-opted to peddle a subpar product or scam. A service chatbot may misunderstand a prompt and fail to process requests as intended. These harms may not immediately or dramatically change people’s lives, but their impacts could nonetheless harm people’s well-being in appreciable ways.

Today’s risk-based AI governance frameworks would likely deem these types of scenarios as “low risk.” For instance, the EU AI Act, which recently entered into force, would consider these situations to pose “limited risks” and receive lower levels of scrutiny compared to “high-risk” contexts (such as when an AI system is used to determine a family’s “riskiness” for child welfare concerns). While high-risk scenarios are prioritized for the most thorough risk management process, low-risk instances are generally deprioritized, subject to more relaxed regulatory standards and a higher tolerance for such risks to manifest.

At first glance — and particularly when considering the severity of discrete events — AI systems that post these sorts of lower risks may appear relatively inconsequential. But if we consider their aggregated effects on the societal level and their compounded effects over time, these risks may not be as mundane as they may initially appear. To safeguard us from the risks of AI, AI practitioners and policymakers must give serious attention to the prevalence of AI’s everyday harms.

Emerging Frameworks Prioritize Perilous Scenarios Over Everyday Realities

To effectively apply safeguards to AI, risk governance frameworks matter. They not only define what risks of AI are pertinent and urgent, but also what are trivial and permissible. Their metrics for prioritization shape how practitioners and policymakers triage their risk management processes, regulatory standards, and tolerance levels. 

From the EU AI Act to the AI Risk Management Framework by the National Institute of Standards and Technology (NIST), regulatory frameworks call for joint consideration of the severity and probability of harm in determining their priority level. In the EU AI Act, for example, “high-risk” refers to scenarios that can cause considerable harm and are somewhat probable in the present and foreseeable future — such as the risks of discrimination by AI systems that make decisions for job hiring, determine loaning decisions, or predict the likelihood of committing a crime.

As these frameworks determine severity levels on the basis of individual occurrences or events, seemingly mundane risks of AI often fall through the cracks. For instance, these frameworks are likely to deem the risks of semantic errors in customer service chatbots as trivial, given that a single incident of such an error is unlikely to have consequential effects on people’s lives. Yet this overlooks how the aggregated harms of such errors may in some cases be just as –– if not more –– severe than what these frameworks deem as “high risk.”

Meanwhile, commercial providers that develop frontier AI technologies, such as OpenAI, Anthropic, and Google Deepmind, are beginning to put greater emphasis on severity over probability. For instance, Anthropic’s Responsible Scaling Policy (RSP) and OpenAI’s Preparedness Framework focus on mitigating risks associated with doomsday scenarios where AI contributes to existential risks, such as pandemics and nuclear wars.  

These frameworks may lead frontier AI developers to overlook risks that are not necessarily catastrophic but nonetheless consequential, especially when viewed in the aggregate. For instance, the risks of cheap fakes –– manipulated content that doesn’t appear realistic yet can still sway public opinions –– would seem to fall under the lowest risk level of OpenAI’s framework despite their ability to distort the perceptions and potential actions of many people. 

The increasing emphasis on the severity level of discrete situations, especially when focused on hypothetical scenarios, risks diverting attention from some of AI’s more common harms and leaving them without adequate investment from practitioners and policymakers.

The Everyday Harms of AI Systems (That We Already Know)

Many of the risks posed by commonplace AI systems like voice assistants, recommender systems, or translation tools may not seem particularly acute when considering discrete encounters on an individual level. However, when considering their aggregated effects on the societal level and their compounded effects over time, the consequences of these risks are much more significant.

1. Linkage Errors Across Databases

Many contemporary AI systems draw on, write to, or otherwise work across multiple systems and databases. This is increasingly true as organizations adopt techniques like retrieval augmented generation (RAG) in order to mitigate risks such as model hallucination. Linkage errors — or missed links between records that relate to the same person or false links between unrelated records — have long proven prevalent in data processing and can have considerable downstream consequences. 

Take, for instance, an AI system responsible for processing hospital admission records. If linkage errors occur within such an AI system, a patient could fail to receive alerts about relevant updates to their case in a timely manner, leading them to miss out on important medical care or be overcharged for services and receive staggering medical bills. Yet, such linkage errors are not necessarily considered to be high risk.

Linkage errors also occur at the population level, where they lead to misclassification and measurement errors in marginalized populations. This is a known problem for organizations like the U.S. Census Bureau and systems like electronic health records (EHRs), and such errors have been shown to disproportionately affect undocumented immigrants, incarcerated people, and unhoused people. When missing or erroneous links go unnoticed in large-scale analyses, decision-makers may unknowingly deprioritize the interests of already marginalized groups. For instance, as undocumented immigrants are often absent or misclassified in digital records, the welfare needs in immigrant neighborhoods may be overlooked in municipal planning over time. Systems most likely to suffer from such errors may not be explicitly classified as higher risk, but the impact of these errors can be profound.

2. Inaccurate Information Retrieval and Summarization

When AI systems serve the function of information retrieval and summarization, semantic errors that misinterpret meanings are commonplace. These errors can occur when systems misunderstand the information they process, or perceive spurious patterns that are misleading or irrelevant to the task (sometimes known as hallucinations). As a result, these systems may misrepresent meanings in their outputs, or create outputs that are nonsensical or altogether inaccurate.

Such errors can significantly undermine people’s trust in the deployers of these systems –– from commercial actors to public services to news providers. Take, for example, an AI-powered sales assistant that gave misleading advice on an airline’s refund policies, contradicting the company’s policies. Scenarios like this can affect both consumers and businesses negatively. Not only could an incident like this lead to lengthy (and oftentimes legal) disputes, but the repeated occurrence of such errors can erode an organization’s reputation and cause significant financial losses for both consumers and businesses over time.

Furthermore, semantic errors can transform into acute risks when the information they present ends up playing a role in informing consequential decisions, from border control to legal verdicts. This could occur when a general-purpose model, which may not have been designed or tested for deployments in high-risk scenarios, is integrated into decision-making systems.

This risk is distinct from the errors that occur when an AI system explicitly makes or assists with a highly consequential decision — say, when it recommends whether a job candidate should be hired or not. Instead, these are errors that might occur when, say, an AI system that summarizes an applicant’s resume inaccurately interprets these documents. Even though the summarization system itself may not directly determine hiring decisions, the inaccurate summarization it provides could lead human decision-makers to make ill-informed judgments.

3. Reduced Visibility in Platformized Environment 

Recommender systems have frequently been shown to reduce the visibility of online content from marginalized groups. As online platforms that leverage such systems increasingly mediate everyday facets of people’s lives, visibility reduction risks deterring certain people from fully participating in public culture and accessing economic opportunities.

Visibility reductions generally occur when recommender systems make a prediction that a piece of content violates a platform’s policies, which then triggers content moderation interventions. If a piece of content is erroneously predicted to violate a policy even if it does not, it is then likely to be unjustly downranked –– or outright banned –– in platformized environments. 

Such interventions are colloquially known as “shadowbanning,” when users are “uninformed of the nature, reason, and extent of content moderation.” For users who depend on visibility and reach for their careers and income, such interventions can cause financial harm. If errors are distributed disproportionately across a population, this can lead to discriminatory or otherwise harmful outcomes: recommender systems have been reported to disproportionately reduce the visibility of content by creators who are women, black, queer, plus-sized, transgender, and/or from the majority world. 

On a societal level, communities whose visibility is repeatedly reduced also find themselves systematically excluded from public conversations on the Internet. Individually, people from marginalized backgrounds often experience feelings of sadness and isolation when they cannot connect with people like themselves on the Internet. Even if recommender systems are not operating in a domain that has been characterized as high-risk, these effects can compound significantly over time.

4. Quality-of-Service Harms

Errors in AI applications lower the quality of service for chatbots and voice assistants, which are increasingly becoming people’s gateway to all sorts of services, from customer support to public service assistance. In addition to simple linguistic errors like grammar, spelling, or punctuation mistakes, semantic errors can lead to major misunderstandings of meanings. Previous research has shown that these systems perform considerably less well for non-English speakers, people of color, people who speak English with a “non-standard” accent, and non-native English speakers.

The impact of high error rates is a form of quality-of-service harm. At a minimum, repeated encounters with such errors mean marginalized users have to expend extra effort and time to reap the benefits of AI applications. At worst, the low quality could prevent people from taking advantage of AI-powered products or important opportunities they mediate. Oftentimes, this deters marginalized populations from accessing services that are set out to empower them in the first place. This sort of error will be critical to keep in mind as a growing number of public services are turning to chatbots and voice assistants as their primary –– or even default –– interfaces.

5. Reputational Harms

From recommender systems to generative models, AI systems pose the risks of depicting regular people in undesirable or harmful ways. Particularly when the depictions are not legally protected under anti-discrimination laws, these grey-area representational harms can end up falling through the cracks.

For instance, when someone searches for information about an individual through a search engine, the first few search results can play a crucial role in determining impressions of the person. When a search for a certain name leads to criminal history and mugshot websites, it paints a negative picture. Since there is no legal requirement for this information to be accurate or timely, the persistence of inaccurate or out-of-date information risks wrongly harming people’s reputations, which can indirectly affect their access to important opportunities. 

Meanwhile, the wide availability of generative models makes it faster and easier to abuse the likeness of others without their consent. Anyone with photos on the Internet may be subjected to the misuse of AI likeness. For instance, to cut down on the cost of paying for models, digital advertisers may instead create AI likenesses based on images of people found online –– oftentimes without permission. Beyond the lack of financial compensation to real models who would otherwise be hired, and to those whose faces and voices are leveraged without consent, such likenesses may also cause harm by portraying people in an undesirable light. For instance, one’s likeness may be manipulated to discuss deeply sensitive matters, as in the abuse of AI likeness to sell health supplements. These likenesses don’t even have to be photo-realistic to cause harm; they just have to be realistic enough. The misuse of AI likeness often occurs repeatedly for the same people and disproportionately falls on women. 

Despite the prevalence of such harms, presently, it can be difficult for everyday people to track down evidence of such harm, and even when adequate interventions do happen, they often only take place long after the harm has already occurred.

Beyond Severity: Rethinking the Goalposts of AI Risk Governance

Given how risk-based frameworks help shape the priorities of both policymakers and practitioners on “high risk” scenarios, AI’s everyday harms are increasingly likely to fall through the cracks. Practitioners and policymakers alike are shifting focus to severe scenarios, with frontier AI developers putting even greater emphasis on severity over probability and frequency.

To effectively and responsibly triage time and resources for AI risk governance, we must move beyond the view that puts greater and greater distance between “high-risk” and “low-risk” scenarios; severity should not be the only metric for prioritization. A more pragmatic orientation of risk governance must also take the prevalence of risks into serious consideration. From shadowbanning on social media platforms to low-quality chatbot service for non-English speakers, AI risk governance frameworks must encourage due attention to the everyday risks of AI that are already creating concrete harms.

While these so-called “lower-risk” scenarios may not appear severe as discrete events, their aggregated impact on the societal level and their immediacy mean that organizations should nevertheless invest in mitigating their risks. 

To prepare for the everyday risks of AI systems, practitioners must be ready to act on their negative residual risks, or risks that remain unmitigated despite the safety measures in place. To do so, AI practitioners need safety infrastructures that can systematically monitor and rapidly respond to these risks as they manifest as harm. Policymakers can even mandate that such infrastructures should always accompany the deployment of AI systems.

For instance, to address semantic errors in chatbot services, organizations may consider post-deployment monitoring, with particular attention to scenarios where users get stuck or encounter a failure mode. When these scenarios take place, organizations can immediately connect these users to human operators for troubleshooting. Not only will this make it possible for organizations to address incidents of harm in a more timely manner, but over time, they will also be able to better anticipate and account for potential vulnerabilities once they have a better understanding of how such harms manifest on the ground.

The everyday risks of AI should not be an afterthought; they deserve far greater priority than they currently receive. Presently, AI’s everyday harms are disproportionately affecting the most vulnerable populations, from unhoused people to non-English speakers, to formerly incarcerated people. Framing harms as “low-risk” means neglecting the harms that AI systems continue to inflict on these populations on a day-to-day basis.