You’re Securing Your Code. But Are You Securing the AI Inside It?
Modern applications don’t just run on code anymore 2026-6-24 09:2:30 Author: checkmarx.com(查看原文) 阅读量:0 收藏

Modern applications don’t just run on code anymore. They run on models, agents, embeddings, datasets, and autonomous tools like MCP servers. Developers are pulling pre-trained LLMs from Hugging Face, integrating open-source agent frameworks, and wiring up AI pipelines faster than security teams can track them. Unlike rogue npm packages, these components don’t show up cleanly in your existing dependency graph. 

For AppSec teams, this creates two core problems:  volume and visibility. 

  • Volume. AI is speeding up development output, but AI-generated code produces 1.7x more vulnerabilities than human-written code. More code means more findings per scan, growing backlogs, and a security team forced to choose between rigor and velocity.  
  • Visibility. The AI components powering development – models, datasets, MCP servers, agent frameworks, and prompt libraries – are entering codebases without formal review or governance. Most organizations don’t know which AI assets they’re running, where they’re embedded, or what risk they carry. That gap has consequences: over 75% of organizations have experienced a software supply chain attack in the last year, because every new AI component introduces a potential attack path.  

As development speeds up, these unmanaged AI components accumulate faster than security teams can evaluate them, pushing AppSec programs beyond the limits they were designed for.  

The Shadow AI Problem in Your Software Supply Chain 

This gap exists because traditional AppSec was built for a world where the building blocks of software were code, libraries, and configurations. That world still exists, but now it runs alongside a parallel supply chain of AI components that most tools can’t see. 

Today, a typical AI-enabled application might include a fine-tuned LLM pulled from a public model hub, an agent framework that invokes external tools autonomously, an MCP server connecting the app to live data sources, embeddings generated from sensitive internal documents, and system prompts hardcoded in config files. 

None of these appear in a standard SBOM and only a few get flagged in a standard code scan. Each introduces distinct risks, from model poisoning and unverified weights to unsafe autonomous tool invocation and exposed datasets. 

Three forces are making this worse: 

  • Visibility is breaking down. Most organizations can’t fully inventory which AI assets are in use, where they live, or what risk they introduce. It mirrors the early days of open-source governance, but with faster adoption and higher-stakes components. 
  • The toolchain has expanded. Every model, dataset, MCP server, API, and open-source dependency creates a potential attack path. This growing web of dependencies increases both complexity and exposure.  
  • Shadow AI is the new shadow IT. Just as shadow IT created governance blind spots in the cloud era, developers and DevOps teams are adopting AI tools, models, and plugins without formal security review, often because no review process exists yet. 

The attack surface isn’t theoretical. The EU AI Act, ISO 42001, and NIST AI RMF now treat AI component governance as a compliance requirement. If your AppSec program can’t answer what AI is in your softwarewhere it is, and what it does, you have a gap. 

Why Existing Approaches Fall Short 

Several vendors have taken a swing at AI security, but the results don’t quite land for AppSec teams.  

  • Cloud-posture tools that view AI through the lens of services and infrastructure exposure. These tools are useful, but they miss what’s embedded directly in code and configuration.  
  • Artifact-level scanners review model files for malicious payloads, but don’t provide visibility into how those models are wired into your applications in the first place. 
  • SCA extensions that identify open-source LLM dependencies in package manifests, but don’t understand agent frameworks, MCP servers, embedded prompts, or dataset references. 

Most importantly, tools that rely on AI inference to detect AI introduce the exact kind of non-determinism that worries auditors. If your compliance report is exclusively based on probabilistic detection, it isn’t audit-ready. 

To close this gap, you need a fundamentally different approach to discovery.  

Deterministic Discovery: Seeing What’s Actually There 

Checkmarx AI Supply Chain Security takes a different approach: code-first, deterministic detection with high-fidelity scanning. Instead of inferring the presence of AI assets, it reads them directly from imports, manifests, file paths, and configuration – the same signals a developer would traditionally follow to understand what a codebase depends on. 

This means: 

  • LLMs and model references are identified by their identifiers in code and config, not guessed from patterns 
  • Agent frameworks are detected from imports and initialization code 
  • MCP servers are discovered from configuration and integration points 
  • Datasets and embeddings are traced from references in source and manifests 
  • System prompts are surfaced from hardcoded strings and config files 

The result is a complete, auditable AI asset inventory, not a probabilistic best guess. Every finding can be traced back to a specific line of code or configuration entry, which matters when you’re presenting results to a compliance auditor or explaining a finding to a developer. 

Risk Assessment Built for AI-Specific Threats 

Visibility helps close the gap, but it doesn’t solve the volume problem. More assets mean more potential risks, more findings, and more decisions – now extending across an entirely new class of components.  

Once you know what AI assets exist, you still need to understand what risks they introduce and how to address them at scale. 

Checkmarx AI Supply Chain Security assesses AI-specific supply chain risks that traditional AppSec tools weren’t built to find, including: 

  • Model poisoning and unverified weights: Models pulled from public sources without integrity verification can carry malicious payloads or backdoors introduced during training. The LLM Security scanner identifies ML artifacts (PyTorch files, GGUF, H5, and othersand evaluates them for deserialization and execution risks. 
  • Unpinned model versions: Floating model references (the AI equivalent of * in a package manifest) let upstream updates silently change your application’s behavior. Version pinning is enforced as a policy requirement. 
  • Unsafe autonomous agents: Agents that invoke external tools without proper scope constraints create execution risks that are unique to AI systems. These are surfaced and assessed as part of the asset inventory.  
  • Exposed datasets and embeddings: Datasets used for fine-tuning or RAG pipelines can expose sensitive internal information if they’re not properly scoped. Dataset references are tracked as first-class assets with their own risk profiles.  
  • Dataset exposure and license violations: Open-source models and datasets carry licensing obligations that differ significantly from traditional software licenses. AI asset metadata includes license information, enabling policy enforcement before non-compliant components ship. 

The Same Workflows. Extended To Cover AI. 

Adding a new security tool usually means adding friction: a new platform to learn, a new set of alerts to triage, a new reporting workflow to maintain in parallel with everything else. This is where AppSec integration matters. 

Checkmarx AI Supply Chain Security is built directly into Checkmarx One, the same platform where you’re already managing vulnerabilities, running SAST and SCA, enforcing policies, and generating compliance reports. There is no separate product or parallel workflow to manage. 

In practice, that means: 

  • AI components appear alongside traditional findings in the same dashboards, with the same triage workflows. 
  • Policy enforcement happens in pull requests and CI/CD, the same way you gate on open-source vulnerabilities or SAST findings today. You can block unapproved models, flag unsafe agents, or require version pinning without writing custom automation. 
  • AI-BOM generation means that AI assets appear in the same Bill of Materials as your OSS dependencies, with origins, licenses, dependencies, and risk metadata attached. 
  • ASPM workflows including risk orchestration, analytics, dashboards, extend naturally to cover AI components without separate instrumentation. 
  • API and CLI support lets AI scanning drop into existing pipeline automation without new integrations. 

The goal is accountability without friction. You’re not replacing your AppSec workflow; you’re just extending it to cover a new class of components. 

Here’s What Your AppSec Team Gets 

The result is a practical extension of your existing AppSec program. 

  • Complete AI asset inventory across the enterprise. Every LLM, agent framework, MCP server, dataset, embedding, and system prompt that exists in your codebase is surfaced deterministically from code and configuration, compiled into a searchable, auditable catalog. 
  • AI-specific risk assessment. Model poisoning, unverified weights, unsafe agents, exposed datasets, unpinned versions are detected with evidence-backed findings and actionable remediation guidance. 
  • Regulatory readiness. AI-BOMs are aligned with compliance posture tracking against EU AI Act, ISO 42001, NIST AI RMF, and OWASP LLM Top 10, and ready to export when an auditor asks. 
  • Policy enforcement in the developer workflow. Approved model lists, framework allowlists, version pinning requirements are enforced in pull requests and CI/CD, not after the fact. 
  • No new platform overhead. Everything lives in Checkmarx One, using the same permissions, dashboards, and reporting your team already relies on.  

Shadow AI in the SDLC: A Practitioner Panel on Visibility, Risk, and the Road to Governed AI

Upcoming Live Webinar | 9 July 10:00AM EDT

The Bottom Line for AppSec 

AI components have changed what your software is made of and are now part of the software supply chain. The question isn’t whether they exist, but whether your AppSec program can handle the visibility and volume that they introduce.   

Governing the AI supply chain isn’t optional anymore. Your AppSec program needs the visibility, tooling, and workflow integration to keep up. Checkmarx AI Supply Chain Security is built for that. Deterministic discovery, evidence-backed risk assessment, and end-to-end governance built directly into developer workflows. It integrates with your existing pipelines, giving you a unified risk picture across code, dependencies, models, and runtime environments without adding another silo to manage. 

See It in Action 

If this gap sounds familiar, it likely already exists in your organization’s environment. The good news?  It’s solvable without disrupting the AppSec program you’ve already built. 

See it in action

Checkmarx AI Supply Chain Security

A Gartner® Magic Quadrant Leader™

A Forrester Wave Leader™

SOC 2 Type II Certified

Tags:

Agentic AI

AI-Generated Code

Open-Source Supply Chain

shadow ai

Software Supply Chain Security


文章来源: https://checkmarx.com/blog/youre-securing-your-code-but-are-you-securing-the-ai-inside-it/
如有侵权请联系:admin#unsafe.sh