Security leaders and developers alike are already acutely aware that AI coding assistants and agentic agents can introduce vulnerabilities into the code they generate. A recent study unveiled another critical concern to keep them up at night — LLMs used for making iterative code improvements may introduce new vulnerabilities over time, even when explicitly asked to make code more secure. 

Researchers from the University of San Francisco, the Vector Institute for Artificial Intelligence in Toronto and the University of Massachusetts Boston analyzed 400 code samples across 40 rounds of ‘improvements’ using four prompting strategies — one of which explicitly asked the LLM to improve security or fix vulnerabilities — and found a 37.6% increase in critical vulnerabilities after just five iterations. 

This counterintuitive problem, which the authors refer to in the report’s title as a ‘paradox’, underscores once again the absolute need for fully trained, skilled human developers to maintain oversight of the development loop, even when AI tools are doing much of the work. 

The study offers several recommendations to help organizations mitigate the risks of what the authors call ‘feedback loop security degradation’, stressing the importance of human-AI collaboration, with human developers taking a hand regularly (and literally) in the process. However, those recommendations also hinge on developers having a medium-to-high level of security proficiency, which is an area where many developers fall short. It’s up to organizations to ensure that developers possess current, verified security skills to work effectively in tandem with AI assistants and agents. 

Vulnerabilities Rise With Each LLM Iteration 

LLMs have been a boon for developers since OpenAI’s ChatGPT was publicly released in November 2022, followed by other AI models. Developers were quick to utilize the tools, which significantly increased productivity for overtaxed development teams. However, that productivity boost came with security concerns, such as AI models trained on flawed code from internal or publicly available repositories. Those models introduced vulnerabilities that sometimes spread throughout the entire software ecosystem. 

One way to address the problem was by using LLMs to make iterative improvements to code-level security during the development process, under the assumption that LLMs, given the job of correcting mistakes, would amend them. The study, however, turns that assumption on its head. Although previous studies (and extensive real-world experience, including our own data) have demonstrated that an LLM can introduce vulnerabilities in the code it generates, this study went a step further, finding that iterative refinement of code can introduce new errors. For example, the study details the ‘inverse phenomenon’ of how a tool designed to fix vulnerable code via iterative feedback can actually degrade code, even when the code is initially secure. As an iteration chain — a sequence of iterations without human intervention — grows, the rate of vulnerability introduction also rises with it.  

The security degradation introduced in the feedback loop raises troubling questions for developers, tool designers and AI safety researchers. The answer to those questions, the authors write, involves human intervention. Developers, for instance, must maintain control of the development process, viewing AI as a collaborative assistant rather than an autonomous tool. Designers are required to incorporate security features into their tools to detect potential vulnerabilities and provide alerts when they are identified. Additionally, safety researchers must develop new mechanisms, including automated tools, that identify problematic code to prevent security degradation.  

The authors of the study offer five steps toward mitigating security degradation when using AI tools: 

Require developer reviews between iterations. This step would draw on human expertise as the first line of defense, providing a level of quality control that can’t be automated.  

Limit consecutive LLM iterations. As vulnerabilities become more common later in an iteration chain, organizations should allow no more than three LLM-only iterations before resetting the chain. 

Review each iteration. Leveraging both human expertise and automated tools, organizations should check their security at each step, rather than waiting until the end of a sequence of iterations. 

Apply conventional static analysis tools between iterations. Be sure to use these tools as complements to, rather than replacements for, human expertise. 

Monitor code complexity. The study found that the likelihood of new vulnerabilities increases with the complexity of the code, so human reviewers need to be alert whenever code complexity rises.  

The common thread in these recommendations is the requirement for human expertise, which is anything but guaranteed. Software engineers typically receive very little security upskilling, if any at all, and have traditionally focused on quickly creating applications, upgrades and services while letting security teams chase after any pesky flaws later on. With AI tools accelerating the pace of DevOps environments, organizations must equip developers with the requisite skills to ensure secure code throughout the software development life cycle (SDLC) if they want to maintain security. To achieve this, organizations must implement ongoing educational programs that provide developers with the necessary skills. 

Skills Developers Must Have to Keep AI in Check 

Forward-thinking organizations are working with developers in applying a security-first mindset to the SDLC, in line with the goals of the U.S. Cybersecurity and Infrastructure Security Agency’s (CISA’s) Secure by Design initiative. This includes a continuous program of agile, hands-on upskilling in sessions designed to meet developers’ needs. For example, training is tailored to the work they do in the programming languages they use, while being available on a schedule that fits their busy workdays. Better still, the security proficiency of humans and their AI coding assistants should be benchmarked, with security leaders able to access data-driven insights on both developer security proficiency and the security accuracy of any commits made with the assistance of AI tooling and agents. Would it not be beneficial to monitor who used what to better manage code review, or verify when we know a particular LLM is failing at specific tasks or vulnerability classes? 

An effective upskilling program not only helps ensure that developers can create secure code, but also that they are equipped to review AI-generated code, identifying and correcting flaws as they appear — whether they first occur in generated AI code or later on during iterative security ‘improvements’.  

The recent study emphasizes what was already becoming clear — direct human oversight is essential to secure code, especially as AI tools become more pervasive. It is at the heart of cybersecurity in an increasingly distributed computing ecosystem. The problems resulting from iterative code improvements can’t be solved with a prompt, as proved by LLMs that introduce security vulnerabilities even after being expressly prompted to fix vulnerabilities.  

Even in this new era of AI-generated coding, skilled human supervision remains essential. CISOs must prioritize upskilling programs that could equip their critical human workforce with those skills.