Project Description

The City of San José, CA is a pioneering municipality in acquiring and implementing artificial intelligence (AI) systems and services for the benefits of employees and residents. As the founding partner of the GovAI Coalition, the City leads the Coalition in proposing adaptable, standardized systems for ensuring safety by thoroughly assessing and mitigating AI risks. San José wants to create a systematic, standardized process that is easily adaptable for all GovAI coalition partners to conduct AI Risk assessment within their respective procurement processes. Currently, there is no comprehensive framework on how to do so.

Project Aims

This document proposes an AI system risk assessment framework, in a systematic and adaptable manner, for various use cases during the software lifecycle such as: procurement, annual software audits, and software department transfers. We propose a standardized method to evaluate risk of AI systems with concrete indicators to be used by the City of San José, CA’s Information Technology department and GovAI Coalition members. Our AI Risk Assessment Framework (RAF) is vendor-agnostic and can be used within the procurement process and post-procurement.

Methodology

We reviewed the City of San José's current documents on AI Risk Review, specifically the GovAI Coalition AI Policy Manual and the Generative AI Guidelines,. We also evaluated federal policy on AI risk management, such as the National Institutes of Standards and Technology (NIST) Generative AI Risk Management Framework (RMF) and existing public frameworks like the University of California Berkeley AI Risk-Management Standards Profile for General-Purpose AI Systems and Foundation Models and the Massachusetts Institute of Technology (MIT) CSAIL AI Risk Repository. For alignment with our local municipality, we conducted case studies on existing state policy and legislation, such as the recently passed SB1047 (Safe and Secure Innovation for Frontier Artificial Intelligence Models Act). We highlighted discrepancies between existing documents and paid particular attention to how risk levels and acceptable risks for AI systems are defined in current documents (e.g. personally identifiable information, opt-out policies, predicted harm) and other AI risk management frameworks. We then defined a systematic, interactive structure for AI risk assessment, proposed a risk aggregation step, offer suggestions on the AI Risk Review process, and provided a brief, educational summary for City employees using AI systems.

Project Impact and Future Work

The AI Risk Assessment Framework for San José, CA and the GovAI Coalition, represents a significant step to ensuring that the acquisition, deployment, and use of AI systems within City operations are conducted in a manner that is ethical, transparent, and aligned with the community's values. This document implements a standardized benchmark for AI risk assessment, and the proposed policy changes highlight the City's Commitment to leading by example in the ethical use of AI technologies. We recommend that the City of San José, CA, implement the AI Risk Assessment Framework through a phased approach, beginning with the establishment of the AI Auditing Sub-Committee and an integration of the AI risk assessment models into existing technology review processes. Additionally, the City should update existing policies to reflect the proposed changes to better solidify its stance on robust, standardized AI risk assessment and mitigation.

Contributors