CDT Comments to NIST on its Assignments Under the Executive Order Concerning Artificial Intelligence

CDT submitted comments to the National Institute of Standards and Technology’s request for information on its assignments under the Executive Order Concerning Artificial Intelligence, with a particular focus on questions posed about the development of guidelines, standards, and best practices related to generative AI risk management and AI evaluation.

Our feedback is grounded in our underlying assessment that risk management of generative AI should be considered a component of general AI governance practices rather than presenting a standalone set of risks that demand separate frameworks. To be sure, generative AI may in some cases give rise to different risks than other AI systems. But too narrowly focusing on generative AI at the expense of paying due attention to the building blocks of AI risk management could encourage piecemeal and fractured organizational approaches that are more likely to overlook key risks and lead to incomplete risk mitigation. A common set of concepts, approaches, and infrastructure for AI risk management is needed to lay the foundation for effectively addressing generative AI-specific risks and harms.

Our comments also urge NIST to ensure that AI evaluation and benchmarking methods and metrics, including those specific to generative AI, are rigorous and scientifically grounded, and encourage the development of more effective benchmarks in non-English languages. Measurement validity and inclusivity is particularly important when it comes to AI, since consequential decisions about the launch and use of advanced systems will increasingly be informed by results of these measurements — and improperly scoped measurements could lead to faulty decisions that threaten people’s safety, access to opportunity, and well-being. We also highlight the central role that documentation of models and systems must place in risk management of both traditional and generative AI systems.

Read the comments.