CISA Enhances Secure by Design Strategy with AI Red Teaming for Critical Infrastructure Protection

CISA Enhances Secure by Design Strategy with AI Red Teaming for Critical Infrastructure Protection
2024-11-27 20:30:46 Author: cyble.com(查看原文) 阅读量:3 收藏

Overview

CISA has announced new additions to its Secure by Design initiative with the introduction of advanced fields in artificial intelligence (AI). This plan ensures the safety, security, and reliability of AI systems, especially as they are increasingly integrated into critical infrastructure and public safety applications. One of the most effective ways to evaluate and improve the resilience of AI systems is through the process of AI red teaming, which is an integral part of a broader strategy known as Testing, Evaluation, Validation, and Verification (TEVV).

This approach, backed by decades of experience in software security testing, emphasizes the importance of a Secure by Design methodology and aims to protect against both technical and ethical risks associated with AI deployment. The Cybersecurity and Infrastructure Security Agency (CISA), as the national coordinator for critical infrastructure security, has been at the forefront of promoting the Secure by Design approach in the development and testing of AI systems.

This initiative is designed to ensure that AI technologies are not only functional but also resistant to exploitation and capable of operating safely within complex environments. In a recent blog post by Jonathan Spring, Deputy Chief AI Officer, and Divjot Singh Bawa, Strategic Advisor, CISA emphasizes the importance of integrating AI red teaming into the established framework of software TEVV.

Red teaming, in the context of AI, refers to third-party safety and security evaluations of AI systems. It is part of a broader risk-based approach that includes thorough testing to uncover vulnerabilities and potential points of failure. According to the CISA blog, AI red teaming is essential for identifying weaknesses that could lead to critical failures, whether through physical attacks, cyberattacks, or unforeseen system malfunctions. The goal of AI testing is to predict how an AI system may fail and develop strategies to mitigate such risks.

AI Testing, Evaluation, Validation, and Verification (TEVV)

TEVV, a well-established methodology used for testing software systems, is not just relevant but essential for evaluating AI systems. Despite some misconceptions, AI TEVV should not be seen as entirely distinct from software TEVV. In fact, AI systems are fundamentally software systems, and the principles of TEVV are directly applicable to AI evaluations. This approach is particularly important as AI becomes increasingly integrated into safety-critical sectors like healthcare, transportation, and aerospace.

The TEVV framework is built upon three core components: system test and evaluation, software verification, and software validation. These processes ensure that software, including AI systems, functions as intended, meets safety standards, and performs reliably in diverse conditions. AI systems, like traditional software, must be rigorously tested for both validity (whether the system performs as expected) and reliability (how well the system performs under varying conditions).

One of the common misconceptions about AI systems is that their probabilistic nature — which allows them to adapt to changing inputs and conditions — makes them fundamentally different from traditional software. However, both AI and traditional software systems are inherently probabilistic, as demonstrated by issues like race conditions in software, where seemingly minor changes can lead to critical errors.

The Intersection of Software and AI TEVV

The notion that AI systems require entirely new testing frameworks separate from software TEVV is flawed. While AI systems may introduce new challenges, particularly around their decision-making processes and data-driven behaviors, many of the testing methodologies used in traditional software security remain relevant.

For instance, AI systems must undergo similar testing to ensure they are robust against unexpected inputs, exhibit reliability over time, and operate within secure boundaries. These concepts are not new but have been applied to traditional software for decades, particularly in industries where safety is paramount.

Take, for example, automated braking systems in modern vehicles. These systems rely on AI to interpret sensor data and make split-second decisions in critical situations, such as detecting pedestrians or obstacles. To ensure these systems are safe, engineers must test their robustness under a variety of scenarios, from unexpected road conditions to sensor malfunctions. Similarly, AI systems, regardless of their complexity, must undergo similar evaluations to guarantee their safety and reliability in real-world conditions.

CISA’s Role in Advancing AI Red Teaming and Security

CISA’s leadership in AI red teaming and security testing is crucial as AI becomes more prevalent in critical infrastructure. The agency is a founding member of the newly formed Testing Risks of AI for National Security (TRAINS) Taskforce, which aims to test advanced AI models used in national security and public safety contexts. The taskforce will focus on creating new AI evaluation methods and benchmarks to ensure that AI systems meet national security standards and can be securely deployed.

Moreover, CISA is actively involved in post-deployment AI security testing. This includes penetration testing, vulnerability scanning, and configuration testing for AI systems deployed across both federal and non-federal entities. As AI technologies, especially Large Language Models (LLMs), become more integrated into various sectors, CISA expects an increase in demand for these security testing services.

In addition to its technical efforts, CISA works closely with the National Institute of Standards and Technology (NIST) to develop and refine standards for AI security testing, providing expertise on how to make these standards actionable and effective.

Conclusion

As the field of AI testing continues to evolve, integrating AI red teaming into the existing software TEVV framework offers significant benefits. By adapting traditional software security testing methods to address the unique challenges posed by AI, the testing community can build upon proven strategies while incorporating new tools and methodologies specific to AI evaluation. This streamlined approach helps save time, resources, and effort by avoiding the creation of parallel testing processes that may ultimately yield similar results.

References