Software is a method of communicating human intent to a machine. When developers write software code, they are providing precise instructions to the machine in a language the machine is designed to understand and respond to. For complex tasks, these instructions can become lengthy and difficult to check for correctness and security. Artificial intelligence (AI) offers the alternative possibility of interacting with machines in ways that are native to humans: plain language descriptions of goals, spoken words, and even gestures or references to physical objects visible to both the human and the machine. Because it is so much easier to describe complex goals to an AI system than it is to develop millions of lines of software code, it is not surprising that many people see the possibility that AI systems might consume greater and greater portions of the software world. However, greater reliance on AI systems might expose mission owners to novel risks, necessitating new approaches to test and evaluation.

SEI researchers and others in the software community have spent decades studying the behavior of software systems and their developers. This research has advanced software development and testing practices, increasing our confidence in complex software systems that perform critical functions for society. In contrast, there has been far less opportunity to study and understand the potential failure modes and vulnerabilities of AI systems, and particularly those AI systems that employ large language models (LLMs) to match or exceed human performance at difficult tasks.

In this blog post, we introduce System Theoretic Process Analysis (STPA), a hazard analysis technique uniquely suitable for dealing with the complexity of AI systems. From preventing outages at Google to enhancing safety in aviation and automotive industries, STPA has proven to be a versatile and powerful method for analyzing complex sociotechnical systems. In our work, we have also found that applying STPA clarifies the safety and security objectives of AI systems. Based on our experiences applying it, we describe four specific ways that STPA has reliably provided insights to enhance the safety and security of AI systems.

The Rationale for System Theoretic Process Analysis (STPA)

If we were to treat a system with AI components like any other system, common practice would call for following a systematic analysis process to identify hazards. Hazards are conditions within a system that could lead to mishaps in its operation resulting in death, injury, or damage to equipment. System Theoretic Process Analysis (STPA) is a recent innovation in hazard analysis that stands out as a promising approach for AI systems. The four-step STPA workflow leads the analyst to identify unsafe interactions between the components of complex systems, as illustrated by the basic security-related example in Figure 1. In the example, an LLM agent has access to a sandbox computer and a search engine, which are tools that the LLM can employ to better address user needs. The LLM can use the search engine to retrieve information relevant to a user’s request, and it can write and execute scripts on the sandbox computer to run calculations or generate data plots. However, giving the LLM the ability to autonomously search and execute scripts on the host system potentially exposes the system owner to security risks, as in this example from the Github blog. STPA offers a structured way to define these risks and then identify, and ultimately prevent, the unsafe system interactions that give rise to them.

Figure 1. STPA Steps and LLM Agent with Tools Example

Historically, hazard analysis techniques have focused on identifying and preventing unsafe conditions that arise due to component failures, such as a cracked seal or a valve stuck in the open position. These types of hazards often call for greater redundancy, maintenance, or inspection to reduce the probability of failure. A failure-based accident framework is not a good fit for AI (or software, for that matter), because AI hazards are not the result of the AI component failing in the same way as a seal or a valve might fail. AI hazards arise when fully-functioning programs faithfully follow flawed instructions. Adding redundancy of such components would do nothing to reduce the probability of failure.

STPA posits that, in addition to component failures, complex systems enter hazardous states because of unsafe interactions among imperfectly controlled components. This foundation is a better fit for systems that have software components, including components that rely on AI. Instead of pointing to redundancy as a solution, STPA emphasizes constraining the system interactions to prevent the software and AI components from taking certain normally allowable actions at times when the actions would lead to a hazardous state. Research at MIT comparing STPA and traditional hazard-analysis methods, reported that, “In all of these evaluations, STPA found all the causal scenarios found by the more traditional analyses, but it also identified many more, often software-related and non-failure, scenarios that the traditional methods did not find.” Past SEI research has also applied STPA to analyze the safety and security of software systems. Recently, we have also used this technique to analyze AI systems. Each time we apply STPA to AI systems—even ones in widespread use—we uncover new system behaviors that could lead to hazards.

Introduction to System Theoretic Process Analysis (STPA)

STPA starts by identifying the set of harms, or losses, that system developers must prevent. In Figure 1 above, system developers must prevent a loss of privacy for their customers, which could result in the customers becoming victims of criminal activity. A safe and secure system is one that cannot cause customers to lose control over their personal information.

Next, STPA considers hazards—system-level states or conditions that could cause losses. The example system in Figure 1 could cause a loss of customer privacy if any of its component interactions cause it to become unable to protect the customers’ private information from unauthorized users. The harm-inducing states provide a target for developers. If the system design always maintains its ability to protect customers’ information, then the system cannot cause a loss of customer privacy.

At this point, system theory becomes more prominent. STPA considers the relationships between the components as control loops, which compose the control structure. A control loop specifies the goals of each component and the commands it can issue to other parts of the system to achieve those goals. It also considers the feedback available to the component, enabling it to know when to issue different commands. In Figure 1, the user enters queries to the LLM and reviews its responses. Based on the user queries, the LLM decides whether to search for information and whether to execute scripts on the sandbox computer, each of which produces results that the LLM can use to better address the user’s needs.

This control structure is a powerful lens for viewing safety and security. Designers can use control loops to identify unsafe control actions—combinations of control actions and conditions that would create one of the hazardous states. For example, if the LLM executes a script that enables access to private information and transmits it outside of the session, this could result in it being unable to protect sensitive information.

Finally, given these potentially unsafe commands, STPA prompts designers to ask, what are the scenarios in which the component would issue such a command? For example, what combination of user inputs and other circumstances could lead the LLM to execute commands that it should not? These scenarios form the basis of safety fixes that constrain the commands to operate within a safe envelope for the system.

STPA scenarios can also be applied to system security. In the same way that a safety analysis develops scenarios where a controller in the system might issue unsafe control actions on its own, a security analysis considers how an adversary could exploit these flaws. What if the adversary intentionally tricks the LLM into executing an unsafe script by requesting that the LLM test it before responding?

In sum, safety scenarios point to new requirements that prevent the system from causing hazards, and security scenarios point to new requirements that prevent adversaries from bringing hazards upon the system. If these requirements prevent unsafe control actions from causing the hazards, the system is safe/secure from the losses.

Four Ways STPA Produces Actionable Insights in AI Systems

We discussed above how STPA could contribute to better system safety and security. In this section we describe how STPA reliably produces insights when our team performs hazard analyses of AI systems.

Table 1. Notional Responsibility Table for LLM Agent with Tools Example

Accidents in AI systems can—and have—occurred due to design errors in specifying each of the elements in Table 1. The box below contains examples of each. In all these examples, none of the system components failed—each behaved exactly as designed. Yet the systems were still unsafe because their designs were flawed.

The responsibility table provides an opportunity to evaluate whether the responsibilities of each controller are appropriate. Returning to the example of the LLM agent, Table 1 leads the analyst to consider whether the control actions, process model, and feedback for the LLM controller enable it to fulfill its responsibilities. The first responsibility of never generating code that exposes the system to compromise is unsupportable. To fulfill this responsibility, the LLM’s process model would need a high level of awareness of when generated code is not secure, so that it would correctly determine when not to provide the execute script command because of a security risk. An LLM’s actual process model is limited to probabilistically completing token sequences. Though LLMs are trained to ignore some requests for insecure code, these steps reduce, but do not eliminate, the risk that the LLM will produce and execute a harmful script. Thus, the second responsibility represents a more modest and appropriate goal for the LLM controller, while other system design decisions, such as security constraints for the sandbox computer, are necessary to fully prevent the hazard.

Figure 2: Examples of accidents in AI systems that have occurred due to design errors in specifying each of the elements outlined in Table 1.

By shifting the focus from individual components to the system, STPA provides a framework for identifying and addressing design flaws. We have found that glaring omissions are often revealed by even the simple step of designating which component is responsible for each aspect of safety and then evaluating whether the component has the information inputs and available actions it needs to accomplish its responsibilities.

3. STPA helps developers consider holistic mitigation of risks. Generative AI models can contribute to hundreds of different types of harm, from helping malware coders to promoting violence. To combat these potential harms, AI alignment research seeks to develop better model guardrails—either directly teaching models to refuse harmful requests or adding other components to screen inputs and outputs.

Continuing the example from Figure 1/Table 1, system designers should include alignment tuning of their LLM so that it refuses requests to generate scripts that resemble known patterns of cyberattack. Still, it might not be possible to create an AI system that is simultaneously capable of solving the most difficult problems and incapable of generating harmful content. Alignment tuning can contribute to preventing the hazard, but it cannot accomplish the task on its own. In these cases, STPA steers developers to leverage all the system’s components to prevent the hazards, under the assumption that the behavior of the AI component cannot be fully assured.

Consider the potential mitigations for a security risk, such as the one from the scenario in Figure 1. STPA helps developers consider a wider range of options by revealing ways to adapt the system control structure to reduce or, ideally, eliminate hazards. Table 2 contains some example mitigations grouped according to the DoD’s system safety design order of precedence categories. The categories are ordered from most effective to least effective. While the LLM-centric safety approach would focus on aligning the LLM to prevent it from generating risky commands, STPA suggests a collection of options for preventing the hazard even if the LLM does attempt to run a risky script. The order of precedence first points to architecture choices that eliminate the problematic behavior as the most effective mitigations. Table 2 describes ways to harden the sandbox to prevent the private information from escaping, such as employing and enforcing principles of least privilege. Moving down through the order of precedence categories, developers could consider reducing the risk by limiting the tools available within the sandbox, screening inputs with a guardrail component, and monitoring activity on the sandbox computer to alert security personnel to potential attacks. Even signage and procedures, such as instructions in the LLM system prompt or user warnings, could contribute to a holistic mitigation of this risk. However, the order of precedence presupposes that these mitigations are likely to be the least effective, pushing developers not to rely solely on human intervention to prevent the hazard.

Table 2: Design Order of Precedence and Example Mitigations

Safety testing faces a different problem. Instead of confirming that the system achieves its goals, safety testing must determine which of all possible system behaviors must be avoided. Identifying these behaviors for AI components presents even greater challenges because of the vast space of potential inputs. Modern LLMs can accept up to 10 million tokens representing input text, images, and potentially other modes, such as audio. Autonomous vehicles and robotic systems have even more potential sensors (e.g., light, detection, and ranging LiDAR), further expanding the range of possible inputs.

In addition to the impossibly large space of potential inputs, there is rarely a single expected output. The utility of outputs depends heavily on the system user and context. It is difficult to know where to begin testing AI systems like these, and, as a result, there is an ever-proliferating ecosystem of benchmarks that measure different elements of their performance.

STPA is not a complete solution to these and other challenges inherent in testing AI systems. However, just as STPA enhances safety by limiting the scope of possible losses to those particular to the system, it also helps define the necessary set of safety tests by limiting the scope to the scenarios that produce the hazards particular to the system. The structure of STPA ensures analysts have opportunity to review how each command could result in a hazardous system state, resulting in a potentially large, yet finite, set of scenarios. Developers can hand this list of scenarios off to the test team, who can then select the appropriate test conditions and data to investigate the scenarios and determine whether mitigations are effective.

As illustrated in Table 3 below, STPA clarifies specific security attributes including proper placement of responsibility for that security, holistic risk mitigation, and link to testing. This yields a more complete approach to evaluating and enhancing safety of the notional use case. A secure system, for example, will protect customer privacy based on design decisions taken to protect sensitive customer information. This design ensures that all components work together to prevent a misdirected or rogue LLM from leaking private information, and it identifies the scenarios that testers must examine to confirm that the design will enforce safety constraints.

Table 3. Summary of STPA Benefits on Notional Example of Customer Data Management

Preserving Safety in the Face of Increasing AI Complexity

The long-standing trend in AI—and software generally—is to continually expand capabilities to meet growing user expectations. This generally results in increasing complexity, driving more advanced approaches such as multimodal models, reasoning models, and agentic AI. An unfortunate consequence is that confident assurances of safety and security have become increasingly difficult to make.

We have found that applying STPA provides clarity in defining the safety and security goals of AI systems, yielding valuable design insights, innovative risk mitigation strategies, and improved development of the necessary tests to build assurance. Systems thinking proved effective for addressing the complexity of industrial systems in the past, and, through STPA, it remains an effective approach for managing the complexity of present and future information systems.