Tiexin Guo Senior DevOps Consultant, Amazon Web Services Author | 4th Coffee

Over the past decade, the rise of microservice architectures, the DevOps culture, and popular tools like Docker and Kubernetes have made deploying applications as containers standard practice in the industry, making container security a top priority.

In this article, we will take a deep look at container security: what are the common container vulnerabilities, how to mitigate risks by container security scanning, and what popular container security tools and solutions are available to integrate into the SDLC in a DevSecOps way. Without further ado, let's get started.

An Introduction to Container Vulnerabilities and Container Security Scanning

Container Vulnerabilities

Containers are created using container images, which act as templates. If the images have vulnerabilities, the containers will introduce those vulnerabilities into the production environment.

Container images have a collection of files in them: source code, but also configurations, binaries, and other dependencies, all of which could introduce vulnerabilities. So, what are the common container vulnerabilities? Here are a few:

• Misconfigurations and hard-coded secrets.
• Vulnerabilities in the application code.
• Vulnerabilities brought by insecure libraries or other dependencies that are imported into the images (for example, malicious code could be introduced by a software supply chain attack).

Container Security Scanning

Fortunately, most container vulnerabilities can be mitigated relatively easily via container security scanning, a process that analyzes images and containers for security issues. Continuous container security scanning, or continuous container security, if you will, is a critical part of DevSecOps.

How does container security scanning work? Typically, there are two different methods:

• by analyzing files, packages, and dependencies. For example, by looking at insecure coding practices or misconfiguration, search for patterns that look like hard-coded secrets, and scan the application's dependencies (including third-party libraries and frameworks) against databases of known vulnerabilities.
• by analyzing containers' activity. Unexpected network traffic or a a shell spawned in a container with an attached terminal are both suspicious activities that could cause security concerns.

Next, let's examine these container vulnerabilities—secrets, misconfiguration, vulnerabilities from code, and dependencies—and how different approaches and tools can help mitigate these risks.

Hard-Coded Secrets and Misconfigurations in Docker Images

Misconfigurations and hard-coded secrets reduce security by making the application and potentially the entire infrastructure vulnerable to easy exploits. Attackers can gain access to critical systems and may even escape from within containers to the host machine.

So, at a bare minimum, we need to check for misconfigurations and ensure no hard-coded secrets are present in Docker images.

Now, let's have a look at a few popular tools that can detect hard-coded secrets and misconfiguration.

ggshield

ggshield (requires a GitGuardian account) is a CLI to find more than 350+ types of hardcoded secrets (specific and generic ones) and 70+ Infrastructure-as-Code security misconfigurations.

Simply install it with your package manager (for example, for macOS, you can install ggshield using Homebrew: brew install gitguardian/tap/ggshield), and then the ggshield secret scan docker command is ready to be used to scan local docker images for secrets present in the image's creation process (Dockerfile and build arguments), and in the image's layers' filesystem.

Here's a demo of how it works:

ggshield provides many more functionalities than simply scanning for secrets. For more details, refer to the cheat sheet below (click to get the PDF) or the official documentation here

SecretScanner

Another option is SecretScanner from Deepfence. It is a standalone, open-source tool that retrieves and searches container and host filesystems, matching the contents against approximately 140 secret types (specific only).

Using SecretScanner is simple: pull the docker image for it by running docker pull quay.io/deepfenceio/deepfence_secret_scanner_ce:2.2.0, and we can simply do a docker run to scan container images.

See it in action below:

trivy

Yet another option is trivy , which is a versatile security scanner that can scan for both secrets (but only specific ones) and misconfiguration in container images, file systems, remote git repositories, and more (will touch on this a bit more in the next section). For now, let's focus on its capabilities of secrets and misconfiguration detection.

It can be installed using your package manager (for example, for macOS, you can also use Homebrew to install it: brew install trivy), and you can use it to scan both Docker images and local file systems.

For example, to scan secrets in an image, run trivy image --scanners secret IMAGE_NAME. Below is an example with a clean result:

$ trivy image --scanners secret ironcore864/go-hello-http:0.0.1
2024-06-23T20:02:49+08:00	INFO	Secret scanning is enabled
2024-06-23T20:02:49+08:00	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-06-23T20:02:49+08:00	INFO	Please see also https://aquasecurity.github.io/trivy/v0.52/docs/scanner/secret/#recommendation for faster secret detection

Trivy also provides built-in checks to detect misconfigurations in popular Infrastructure as Code files, such as Docker, Kubernetes, Terraform, CloudFormation, and more. You can even define your own custom checks.

Here's an example to scan a Dockerfile locally: Given the Dockerfile below (building a simple Golang app with Alpine as the base image):

FROM alpine
WORKDIR $GOPATH/src/github.com/ironcore864/go-hello-http
COPY . .
RUN apk add git
RUN go get ./... && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o hello
CMD ["./hello"]
EXPOSE 8080/tcp

Run the following command to check for misconfigurations (as we can see, 4 are found):

$ trivy fs --scanners misconfig .
2024-06-23T20:05:56+08:00	INFO	Misconfiguration scanning is enabled
2024-06-23T20:05:56+08:00	INFO	Detected config files	num=1
...
Failures: 4 (UNKNOWN: 0, LOW: 1, MEDIUM: 1, HIGH: 2, CRITICAL: 0)

MEDIUM: Specify a tag in the 'FROM' statement for image 'alpine'
When using a 'FROM' statement you should use a specific tag to avoid uncontrolled behavior when the image is updated.
See https://avd.aquasec.com/misconfig/ds001

...

HIGH: Specify at least 1 USER command in Dockerfile with non-root user as argument
Running containers with 'root' user can lead to a container escape situation. It is a best practice to run containers as non-root users, which can be done by adding a 'USER' statement to the Dockerfile.
See https://avd.aquasec.com/misconfig/ds002

...

HIGH: '--no-cache' is missed: apk add git
You should use 'apk add' with '--no-cache' to clean package cached data and reduce image size.
See https://avd.aquasec.com/misconfig/ds025

...

LOW: Add HEALTHCHECK instruction in your Dockerfile
You should add HEALTHCHECK instruction in your docker container images to perform the health check on running containers.
See https://avd.aquasec.com/misconfig/ds026

Docker Image Scanning for CVEs

The previous section ensures that there are no hard-coded secrets and that the configurations are spot-on. Next, let's move on to the other files in the Docker image: OS packages, language-specific packages, dependencies, etc.

The aforementioned comprehensive scanner trivy is one of the most popular open-source security scanners. It can also detect known vulnerabilities according to the versions of installed packages. This means that everything inside the image, whether called by your code or not, is scanned.

For example, let's write a simple Python Flask application and build a Docker image out of it, then scan it with trivy.

A very simple Flask hello world app is created:

from flask import Flask

app = Flask(__name__)


@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

Then, we add some dependencies in requirements.txt:

requests==2.19.1
cryptography==3.3.2
flask

As you noticed, some packages are added, but not even used by our Flask app code, simply as a test.

Then we create a Dockerfile to build it:

FROM python:3.9-slim-buster
WORKDIR /app
COPY ./requirements.txt /app
RUN pip install -r requirements.txt
COPY . .
EXPOSE 5000
ENV FLASK_APP=app.py
CMD ["flask", "run", "--host", "0.0.0.0"]

If you wish, you can clone the repository here to get the whole code above.

We build and tag it: docker build . -t ironcore864/python-insecure-app:0.0.1, then scan it with trivy:

$ trivy image --scanners vuln ironcore864/python-insecure-app:0.0.1
2024-06-23T20:50:02+08:00	INFO	Vulnerability scanning is enabled
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pip" version="23.0.1"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="setuptools" version="58.1.0"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="MarkupSafe" version="2.1.5"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="certifi" version="2024.6.2"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="cffi" version="1.16.0"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="chardet" version="3.0.4"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="click" version="8.1.7"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="cryptography" version="3.3.2"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="idna" version="2.7"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pycparser" version="2.22"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="requests" version="2.19.1"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="six" version="1.16.0"
2024-06-23T20:50:03+08:00	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="urllib3" version="1.23"
2024-06-23T20:50:03+08:00	INFO	Detected OS	family="debian" version="10.13"
2024-06-23T20:50:03+08:00	INFO	[debian] Detecting vulnerabilities...	os_version="10" pkg_num=93
2024-06-23T20:50:03+08:00	INFO	Number of language-specific files	num=1
2024-06-23T20:50:03+08:00	INFO	[python-pkg] Detecting vulnerabilities...

ironcore864/python-insecure-app:0.0.1 (debian 10.13)

Total: 208 (UNKNOWN: 3, LOW: 92, MEDIUM: 61, HIGH: 50, CRITICAL: 2)

┌────────────────────────┬─────────────────────┬──────────┬──────────────┬─────────────────────────┬────────────────────────┬──────────────────────────────────────────────────────────────┐
│        Library         │    Vulnerability    │ Severity │    Status    │    Installed Version    │     Fixed Version      │                            Title                             │
├────────────────────────┼─────────────────────┼──────────┼──────────────┼─────────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ apt                    │ CVE-2011-3374       │ LOW      │ affected     │ 1.8.2.3                 │                        │ It was found that apt-key in apt, all versions, do not       │
│                        │                     │          │              │                         │                        │ correctly...                                                 │
│                        │                     │          │              │                         │                        │ https://avd.aquasec.com/nvd/cve-2011-3374                    │
├────────────────────────┼─────────────────────┤          │              ├─────────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ bash                   │ CVE-2019-18276      │          │              │ 5.0-4                   │                        │ bash: when effective UID is not equal to its real UID the... │
│                        │                     │          │              │                         │                        │ https://avd.aquasec.com/nvd/cve-2019-18276                   │
│                        ├─────────────────────┤          │              │                         ├────────────────────────┼──────────────────────────────────────────────────────────────┤
│                        │ TEMP-0841856-B18BAF │          │              │                         │                        │ [Privilege escalation possible to other user than root]      │
│                        │                     │          │              │                         │                        │ https://security-tracker.debian.org/tracker/TEMP-0841856-B1- │
│                        │                     │          │              │                         │                        │ 8BAF                                                         │
├────────────────────────┼─────────────────────┼──────────┼──────────────┼─────────────────────────┼───────

... (truncated)

As you can see, multiple vulnerabilities are found, some of which are from the OS packages and some from the dependencies we added to it. Even though we do not call those packages directly with our code, since image scanning works by scanning the OS packages and dependencies—what's in the image—it finds them all.

Besides trivy, there are a few other tools worth mentioning:

• clair: an open-source application for parsing image contents and reporting vulnerabilities affecting the contents, done via static analysis, not at runtime.
• grype: a vulnerability scanner for container images and filesystems. It works with syft, the powerful SBOM (software bill of materials) tool for container images and filesystems. For more information about SBOMs:

SCA: Software Composition Analysis

Next, let's move on to a slightly different approach to vulnerability scanning: Software Composition Analysis (SCA), which automatically scans and detects vulnerabilities in components and third-party libraries.

While this might sound similar to Docker image scanning, there are key differences. SCA focuses on identifying vulnerabilities in the software dependencies and libraries used within an application, ensuring that all components are secure. In contrast, as we have seen, Docker image scanning examines the entire container image, including the operating system, application code, and configurations.

Let's have a look at one of the SCA tools snyk to get a hands-on feeling about SCA. For macOS users, you can install snyk by running:

brew tap snyk/tap
brew install snyk

If we scan the same app from the previous section:

git clone https://github.com/IronCore864/python-insecure-app.git
cd python-insecure-app
snyk test .

We should get results similar to:

$ snyk test .

Testing ....

Tested 15 dependencies for known issues, found 4 issues, 4 vulnerable paths.


Issues to fix by upgrading dependencies:

  Upgrade [email protected] to [email protected] to fix
  ✗ Uncontrolled Resource Consumption [Low Severity][https://security.snyk.io/vuln/SNYK-PYTHON-CRYPTOGRAPHY-6913422] in [email protected]
    introduced by [email protected]

  Pin [email protected] to [email protected] to fix
  ✗ Cross-site Scripting (XSS) [Medium Severity][https://security.snyk.io/vuln/SNYK-PYTHON-JINJA2-6809379] in [email protected]
    introduced by [email protected] > [email protected]

  Pin [email protected] to [email protected] to fix
  ✗ Improper Removal of Sensitive Information Before Storage or Transfer (new) [Medium Severity][https://security.snyk.io/vuln/SNYK-PYTHON-URLLIB3-7267250] in [email protected]
    introduced by [email protected] > [email protected]

  Pin [email protected] to [email protected] to fix
  ✗ Remote Code Execution (RCE) [High Severity][https://security.snyk.io/vuln/SNYK-PYTHON-WERKZEUG-6808933] in [email protected]
    introduced by [email protected] > [email protected]

...

As we can see, the issues detected by the Docker image scanning in the previous section were also detected by image scanning. Note, however, that image scanning found more issues.

Both Docker image scanning and SCA work by referring to known vulnerability databases, which explains the similar results.

However, whereas Docker image scanning scans everything present inside an image, including OS packages, SCA is only concerned about the dependencies declared in our project (in our case, defined in the requirements.txt), which is a more specific scope.

At the end of the day, vulnerable dependencies (in our case, requests and cryptography) are reported by both Docker image scanning and SCA, but there are key differences:

• When the scan happens in the DevOps cycle: Docker image scanning scans built images. So the scan can only happen once the image is ready, in other words, towards the end of the development cycle (typically in your CI pipeline, just before deploying it in a test environment). SCA, on the other hand, scans the source code, which means SCA is implemented in the early stages of the cycle. The bottom line is that SCA allows us to discover issues earlier.
• What is the real scope of the scan? As we mentioned, SCA is only concerned with how the running code could be compromised, nothing more. But this code will run in a container, which presents its own attack surface. Ultimately, the container image is the blueprint for what will run in the production environment, so it's crucial to detect both the vulnerabilities affecting the container AND the application.

Combining the above pros and cons of both Docker image scanning and SCA, we can see that they provide complementary values at different stages. They both have their place in your automated DevSecOps workflow.

Besides snyk, there is another open-source project osv-scanner that's worth a look at: It's an SCA vulnerability scanner written in Go, which uses the data provided by osv.dev

For more information on osv-scanner (and open-source security in general), read this blog:

Container Runtime Security

So far, we've covered both Docker image scanning and SCA, which happened at the code/image level, with which we can discover the known vulnerabilities and fix them before our apps get deployed. Next, let's cover another type of container security scanning: container runtime security.

Container runtime security, as the name suggests, happens at the runtime level: it takes proactive measures and controls to protect containerized applications during the runtime phase, detecting unexpected behaviour, configuration changes, and attacks in near real-time.

This sounds theoretical, so let's take a look at one concrete example with Falco, an open-source tool designed to detect and alert on abnormal behaviour and potential security threats in real-time.

To quickly try Falco out, prepare a K8s cluster and install it with helm:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
kubectl create namespace falco
helm install falco -n falco --set driver.kind=ebpf --set tty=true falcosecurity/falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.slack.webhookurl=$(base64 --decode <<< "aHR0cHM6Ly9ob29rcy5zbGFjay5jb20vc2VydmljZXMvVDA0QUhTRktMTTgvQjA1SzA3NkgyNlMvV2ZHRGQ5MFFDcENwNnFzNmFKNkV0dEg4") \
--set falcosidekick.config.slack.minimumpriority=notice \
--set falcosidekick.config.customfields="user:changeme"

Note: Be sure to change the last parameter from changeme to your name.

For a more detailed installation guide, see the official documentation here.

Once Falco is up and running, we can simulate suspicious actions to see if the so-called "runtime security" platform can capture them. For As a first example, let's start a container and get a shell. In the real world, an attacker with shell access means the container is already compromised, as the attacker could try all kinds of things.

Start a container by running: kubectl run alpine --image alpine -- sh -c "sleep infinity", then execute the uptimecommandinside it:

kubectl exec -it alpine – sh -c "uptime"

(well, this is a harmless command, but you get the gist).

If we check Falco's output, we should get an alert:

22:09:57.827858816: Notice A shell was spawned in a container with an attached
...
cmdline=sh -c uptime pid=6461 terminal=34816
... (truncated)

Falco's base rules alert you if someone runs an interactive shell into a running container. For more standard rules, see the official documentation. Also, you can write and customize Falco rules by yourself to secure your environment.

Summary

In this blog, we've seen that there are several types of container vulnerabilities: hard-coded secrets, misconfigurations, vulnerabilities in the application code, and vulnerabilities in OS packages and dependencies.

Luckily, most of them can be detected by tools fairly easily:

• ggshield, SecretScanner to scan for secrets
• trivy to scan for misconfigurations
• trivy, clair and grype to scan Docker images for known CVEs
• SCA tools like snyk and osv-scanner to analyze software composition

Besides scanning code and images, we also can enhance our production environment by using container runtime security platforms such as Falco, which detects unexpected behaviour and attacks in real-time with predefined and customizable rules.

As a closing thought, it's worth pointing out that container security scanning has limitations: It's not great for detecting unknown vulnerabilities. If some vulnerabilities are not yet publicly disclosed, or if a new type of attacking behavior just emerged and it's not in the runtime security platform's rule list, they can't be detected.

So, although container security scanning is necessary to secure production environments, it's important to remember that it is one layer of security and not an end-game solution to mitigate all types of threats.

As a follow-up reading, you can refer to the Docker security cheat sheet, to help you understand how to run containers with the right security guardrails from the onset:

I hope you enjoyed this article. See you in the next piece!

*** This is a Security Bloggers Network syndicated blog from GitGuardian Blog - Code Security for the DevOps generation authored by Guest Expert. Read the original post at: https://blog.gitguardian.com/container-security-scanning-vulnerabilities-risks-and-tooling/