As defense and national security organizations consider integrating AI into their operations, many acquisition teams are unsure of where to start. In June, the SEI hosted an AI Acquisition workshop. Invited participants from government, academia, and industry described both the promise and the confusion surrounding AI acquisition, including how to choose the right tools to meet their mission needs. This blog post details practitioner insights from the workshop, including challenges in differentiating AI systems, guidance on when to use AI, and matching AI tools to mission needs.

This workshop was part of the SEI’s year-long National AI Engineering Study to identify progress and challenges in the discipline of AI Engineering. As the U.S. Department of Defense moves to gain advantage from AI systems, AI Engineering is an essential discipline for enabling the acquisition, development, deployment, and maintenance of those systems. The National AI Engineering Study will collect and clarify the highest-impact approaches to AI Engineering to date and will prioritize the most pressing challenges for the near future. In this spirit, the workshop highlighted what acquirers are learning and the challenges they still face.

Some workshop participants shared that they are already realizing benefits from AI, using it to generate code and to triage documents, enabling team members to focus their time and effort in ways that were not previously possible. However, participants reported common challenges that ranged from general to specific, for example, determining which AI tools can support their mission, how to test those tools, and how to identify the provenance of AI-generated information. These challenges show that AI acquisition is not just about picking a tool that looks advanced. It is about choosing tools that meet real operational needs, are trustworthy, and fit within existing systems and workflows.

Challenges of AI in Defense and Government

AI adoption in national security has special challenges that do not appear in commercial settings. For example:

• The risk is higher and the consequences of failure are more serious. A mistake in a commercial chatbot might cause confusion. A mistake in an intelligence summary could lead to a mission failure.
• AI tools must integrate with legacy systems, which may not support modern software.
• Most data used in defense is sensitive or classified. It should be safeguarded at all phases of the AI lifecycle.

Assessing AI as a Solution

AI should not be viewed as a universal solution for every situation. Workshop leaders and attendees shared the following guidelines for evaluating whether and how to use AI:

• Start with a mission need. Choose a solution that addresses the requirement or will improve a specific problem. It may not be an AI-enabled solution.
• Ask how the model works. Avoid systems that function as black boxes. Vendors need to describe the training process of the model, the data it uses, and how it makes decisions.
• Run a pilot before scaling. Start with a small-scale experiment in a real mission setting before issuing a contract, when possible. Use this pilot to refine requirements and contract language, evaluate performance, and manage risk.
• Choose modular systems. Instead of seeking versatile solutions, identify tools that can be added or removed easily. This improves the chances of system effectiveness and prevents being tied to one vendor.
• Build in human oversight. AI systems are dynamic by nature and, along with testing and evaluation efforts, they need continuous monitoring—particularly in higher risk, sensitive, or classified environments.
• Look for trustworthy systems. AI systems are not reliable in the same way traditional software is, and the people interacting with them need to be able to tell when a system is working as intended and when it is not. A trustworthy system provides an experience that matches end-users’ expectations and meets performance metrics.
• Plan for failure. Even high-performing models will make mistakes. AI systems should be designed to be resilient so that they detect and recover from issues.

Matching AI Tools to Mission Needs

The specific mission need should drive the selection of a solution, and improvement from the status quo should determine a solution’s appropriateness. Acquisition teams should make sure that AI systems meet the needs of the operators and that the system will work in the context of their environment. For example, many commercial tools are built for cloud-based systems that assume constant internet access. In contrast, defense environments are often subject to limited connectivity and higher security requirements. Key considerations include:

• Make sure the AI system fits within the existing operating environment. Avoid assuming that infrastructure can be rebuilt from scratch.
• Evaluate the system in the target environment and circumstances before deployment.
• Verify the quality, variance, and source of training data and its applicability to the situation. Low-quality or imbalanced data will reduce model reliability.
• Set up feedback processes. Analysts and operators must be capable of identifying and reporting mistakes so that they can improve the system over time.

Not all AI tools will fit into mission-critical operating processes. Before acquiring any system, teams should understand the existing constraints and the possible consequences of adding a dynamic system. That includes risk management: knowing what could go wrong and planning accordingly.

Data, Training, and Human Oversight

Data serves as the cornerstone of every AI system. Identifying appropriate datasets that are relevant for the specific use case is paramount for the system to be successful. Preparing data for AI systems can be a considerable commitment in time and resources.

It is also necessary to establish a monitoring system to detect and correct undesirable changes in model behavior, collectively referred to as model drift, that may be too subtle for users to notice.

It is essential to remember that AI is unable to assess its own effectiveness or understand the significance of its outputs. People should not put full trust in any system, just as they would not place total trust in a new human operator on day one. This is the reason human engagement is required during all stages of the AI lifecycle, from training to testing to deployment.

Vendor Evaluation and Red Flags

Workshop organizers reported that vendor transparency during acquisition is essential. Teams should avoid working with companies that cannot (or will not) explain how their systems work in basic terms related to the use case. For example, a vendor should be willing and able to discuss the sources of data a tool was trained with, the transformations made to that data, the data it will be able to interact with, and the outputs expected. Vendors do not need to divulge intellectual property to share this level of information. Other red flags include

• limiting access to training data and documentation
• tools described as “too complex to explain”
• lack of independent testing or audit options
• marketing that is overly optimistic or driven by fear of AI’s potential

Even if the acquisition team lacks knowledge about technical details, the vendor should still provide clear information regarding the system’s capabilities and their management of risks. The goal is to confirm that the system is suitable, reliable, and prepared to support real mission needs.

Lessons from Project Linchpin

One of the workshop participants shared lessons learned from Project Linchpin:

• Use modular design. AI systems should be flexible and reusable across different missions.
• Plan for legacy integration. Expect to work with older systems. Replacement is usually not practical.
• Make outputs explainable. Leaders and operators must understand why the system made a specific recommendation.
• Focus on field performance. A model that works in testing might not perform the same way in live missions.
• Manage data bias carefully. Poor training data can create serious risks in sensitive operations.

These points emphasize the importance of testing, transparency, and responsibility in AI programs.

Integrating AI with Purpose

AI will not replace human decision-making; however, AI can enhance and augment the decision making process. AI can assist national security by enabling organizations to make decisions in less time. It can also reduce manual workload and improve awareness in complex environments. However, none of these benefits happen by chance. Teams need to be intentional in their acquisition and integration of AI tools. For optimal outcomes, teams must treat AI like any other essential system: one that requires careful planning, testing, supervising, and strong governance.

Recommendations for the Future of AI in National Security

The future success of AI in national security depends on building a culture that balances innovation with caution and on using adaptive strategies, clear accountability, and continual interaction between humans and AI to achieve mission goals effectively. As we look toward future success, the acquisition community can take the following steps:

• Continue to evolve the Software Acquisition Pathway (SWP). The Department of Defense’s SWP is designed to increase the speed and scale of software acquisition. Adjustments to the SWP to provide a more iterative and risk-aware process for AI systems or systems that include AI components will enhance its effectiveness. We understand that OSD(A&S) is working on an AI-specific subpath to the SWP with a goal of releasing it later this year. That subpath may address these needed improvements.
• Explore technologies. Become familiar with new technologies to understand their capabilities following your organization’s AI guidance. For example, use generative AI for tasks that are very low priority and/or where a human review is expected - summarizing proposals, generating contracts, and developing technical documentation. Individuals need to be careful to avoid sharing private or secret information on public systems and will need to closely check the outputs to avoid sharing false information.
• Advance the discipline of AI Engineering. AI Engineering supports not only developing, integrating, and deploying AI capabilities, but also acquiring AI capabilities. A forthcoming report on the National AI Engineering Study will highlight recommendations for developing requirements for systems, judging the appropriateness of AI systems, and managing risks.