In Part 1, we described the starting position and roadmap for introducing Detection-as-Code in an existing environment, this second part explains how we executed that roadmap.

Some of the content in this part of the series is rather technical. If you are not a technical reader, feel free to skim this part, but do not miss Part 3, which returns to conceptual and strategic topics.

Preparation

The documents associated with a detection, but not directly part of its implementation, were intended to be machine-readable. In practice, however, many were not. The issues ranged from character-encoding problems and indentation errors to random null bytes in files. Furthermore, some files did not comply with the intended schema because keys were missing or incorrect.

This did not cause significant problems in the old process. The documents were written, sometimes in plain-text editors without syntax highlighting, placed on file shares, and left there. In the new setup, however, the repository was intended to become the central single source of truth. All documents therefore needed to be parseable so that their correctness could be validated. Their contents were also required to execute the migration; for example, the metadata documents contained paths to the Atomic Red Team tests.

To systematically correct these documents, we implemented a parsing library containing functions to read the different file types, validate them against a defined schema using strictyaml and jsonschema, and return the parsed results. These functions were used throughout the migration and later in automations.

Using these parsers, we attempted to parse all files and identified issues requiring correction. While some issues could be addressed semi-automatically with scripts, for example by identifying and converting encodings, many had to be corrected manually, file by file.

Because the existing deployment procedures required detection data in platform-specific formats, we needed tooling to generate those formats. For Splunk, this meant an apps directory that could be copied to the Splunk infrastructure.

For Splunk, the process was relatively straightforward. The configuration files for a Splunk app, namely savedsearches.conf, transforms.conf, and macros.conf, could be created by concatenating the corresponding snippets, splunk-searches.conf, splunk-transforms.conf, and splunk-macros.conf, from all active detections assigned to a specific app. Lookup files were copied unchanged to the lookups folder of the relevant app.

In the customer environment, Splunk detections were distributed across different Splunk apps based on their categories. The deployment script therefore iterated through all detection folders in the new repository, determined the target app from the data in meta.yml, and added the relevant configuration snippets and files to that app. Folders containing detections targeting platforms other than Splunk were skipped.

Deployment scripts for the other platforms were created in the same way. They used the new detection repository as input and, without requiring additional parameters, generated the artefacts expected by the existing deployment processes.

We also created a script for Atomic Red Team tests because the framework expects each test to be stored in a separate folder with a specific name. The script built the required path from information in meta.yml and copied the contents of each test file to the required location.

After preparing the current data for the migration and the tooling required to continue deployment after the migration, the next step was to focus on the migration itself. The migration was scripted using Jupyter notebooks, which used the previously implemented parsers and operating-system libraries to copy, move, and edit files. The detailed migration process is covered in the next section.

With the scripted migration and the CLI tools, we performed multiple dry runs. For each dry run, we executed the migration locally on a copy of all data and used the developed CLI tools to regenerate the deployment artefacts, which were expected to match the current data structure. Comparing the generated artefacts with the pre-migration data enabled us to identify and fix open issues, such as incorrect filenames or references in metadata files, until the generated artefacts matched the initial input.

Lastly, before starting the short repository freeze and executing the migration in production, we prioritised completing as many ongoing developments as possible and merging their feature branches into the main branch. This reduced the necessary cleanup work after the migration, as the migration covered only the main branches. Feature branches no longer matched the repository structure after the migration and therefore required their diffs to be migrated manually.

Execution of the Data Restructuring

Because some data had already been stored and tracked in Git repositories for some time, we wanted to preserve its history in full. We therefore needed to perform all moves in a way that Git could trace the history. Git does not store file moves directly; instead, it infers them by comparing file versions. We therefore structured the migration so that Git could recognise moved files rather than treat them as entirely new data. To achieve this, we split the changes into two steps: one that left filenames and paths unchanged while changing only file content, and another that moved or renamed files without changing their contents. This enabled Git to reconstruct the complete change history, and git blame could still show when a line was changed, even when the change had been made in the old structure.

Moving all detection data into the new repository while retaining the old repositories for platform configuration required the histories of all repositories to be connected. For the restructuring, we used a local Git repository as an integration point before pushing the new content back to GitHub. Git allows branches from different repositories to be fetched and merged, even when they have no common history. We therefore added the old repositories as remotes, fetched the relevant branches, and brought their contents into the single local repository as unconnected trees. These branches could then be merged like any other branches in a Git repository. Because this setup is relatively uncommon, the Git CLI requires the ‑‑allow‑unrelated‑histories flag to confirm that the action is intentional. After all restructuring and Git operations, the local repository contained a branch with the new content for each repository. These branches were then pushed back to their respective remotes.

Because each repository’s own history was included in the history of its new branch, we could push the result to the remotes without force-pushing. Where the main branch was protected, we pushed the result as a feature branch and merged it through a pull request on GitHub.

The main branches of all repositories, as well as the file share, had to be frozen from the moment their data was pulled into the local repository and the file-share content was downloaded to the point at which the new contents had been pushed to the relevant repositories and approved through pull requests. Because the migration process was scripted and tested beforehand, the freeze period, including manual verification, could be kept to approximately half a day. With only the main branches frozen, development on feature branches continued during the migration.

The full graph of the restructuring in the local repository is shown below. Each step is explained in the following paragraphs.

Within the local repository, we started with a new empty main branch and initialised it with a README.md file explaining the structure of the new detection repository. We then started building the new structure by adding the metadata documents from the file share. For each detection, we created a folder based on the metadata and placed the document from the file share inside it as meta.yml.

We then migrated the data from an existing Git repository. Starting from the tip of the main branch, we created two branches: one for the data to be moved to the new detection repository and another for the remaining data in the platform-specific repository. All data was distributed between these two branches so that, after the restructuring, each item existed in only one branch. In the branch for the detection repository, files were moved and renamed to match the target structure. As described earlier, we separated renames and content changes into different commits so that Git could reconstruct the history reliably.

For the Splunk repository, we checked out both branches into adjacent working directories. We then removed the existing content from the working directory for the detection-repository branch. The migration script then iterated through the Splunk apps in the working directory for the platform-repository branch and processed their searches, transforms, macros, and lookup files.

For each configuration block, the script attempted to identify the corresponding detection. When a match was found, it removed the block from the source configuration file and moved it to the matching detection folder in the other working directory. For transforms, the associated lookup file was moved together with the configuration block. To preserve the Git history of these moves, we used two commits. First, we copied the complete original configuration file, such as savedsearches.conf, to the target detection folder. After committing this copy, we removed the full file and added only the configuration blocks relevant to the detection in a separate commit. This left Git with sufficient similarity between the old and new files to recognise the move, even though most of the original file had been removed.

The Splunk repository also contained Atomic Red Team test scripts that had to be moved as part of the migration. However, test cases in these files did not follow a sufficiently consistent naming scheme to match them reliably to individual detections. Because the tests were used only manually during development, a complete matching of test cases to detections was not required for the initial restructuring. Instead, we copied each test script unchanged into the folder of every detection associated with the relevant MITRE ATT&CK technique. The resulting duplication was addressed during the post-restructuring cleanup described later.

After moving the data for the detection repository into its working directory and committing the changes in stages, this branch was ready to be merged. Because the main branch, which already contained the metadata documents, and the migration branch both already followed the target structure, the branches could be merged cleanly.

In the working directory for the platform-repository branch, the Splunk app folders then contained only configuration snippets that could not be matched to a detection. In our project, this included two transforms belonging to detections that had already been deleted but whose transforms had been overlooked. We pushed this branch to the original Splunk repository and merged it into the main branch through a pull request.

Restructuring the platform repository itself was outside the scope of the project. However, because it is now maintained solely by the Splunk platform team, a later overhaul can be performed without requiring coordination with the detection engineers.

After migrating the Splunk repository, we followed a similar procedure for the repositories containing data for the other detection platforms.

Post-Restructuring Cleanup

During the restructuring, only the main branches were frozen and development on feature branches continued. However, as these feature branches still had the old structure, the work could no longer be merged. These developments had to be recreated as feature branches in the new repository, after which the old branches could be closed without merging.

A developer working on a detection could continue working on the existing feature branch until it was ready for review. Once ready, the changes had to be moved to the detection repository. In this case, we used the diff shown in the GitHub pull request, which was more efficient than automating cherry-picks or another migration mechanism, and manually recreated the changes on a new feature branch in the detection repository. We then opened a new pull request for review, linked it to the old pull request where relevant context already existed, and closed the old pull request without merging.

The complete Atomic Red Team test script was copied into the folder of every detection covering the same MITRE ATT&CK technique. While this was a reasonable decision to enable the restructuring to proceed, it was not acceptable as a long-term solution. For each detection, all test cases belonging to other detections had to be removed from the test script. Fortunately, this could be partially automated because many test cases roughly followed a naming scheme that referenced the specific detection. The remaining test cases required manual investigation to assign them to a detection and retain them only within that detection.

Migration of remaining distributed data

At this point, all data originating from the file share and the different Git repositories had been organised in the new structure. To fully achieve the original goal of consolidating all detection-related data in GitHub, the final step was to address the data still stored in Excel reports.

Because the Excel reports are required for internal and regulatory processes, they must continue to exist and be kept up to date. However, after the migration, they were generated and updated from the data in GitHub, removing the need for manual edits.

Most of the data in the reports was already present in the metadata documents for the detections. For the additional data in the reports, we extended the metadata schema with the required fields. Afterwards, a script converted the metadata documents to the new schema, and the additional data from the Excel reports was added in a pull request.

In parallel, we developed CLI tools that parse the metadata documents and populate or update the existing Excel reports with data from the repository. This allows up-to-date reports to be generated at any time without manual editing; they always contain the most recent data from GitHub as the single source of truth.

Wrapping Up

We achieved two of the three main goals defined in Part 1.

First, we restructured the existing data to reflect the semantic structure of detections and the way developers interact with them, rather than the structure imposed by the detection platforms. Second, we successfully consolidated all detection-related data in GitHub. No detection-related data remained scattered across different locations. With GitHub as the central source of truth, information could be found easily and changes could be tracked cleanly.

The third goal, improving data quality and building on the new foundation with further automation, is the focus of Part 3. There, we will show how we identified and addressed data-quality issues, the measures taken to maintain a high-quality dataset over time, and the automations enabled by the new foundation.