An exploit

Let's start by taking a short look into what happens during a typical exploit scenario. Everything begins with a vulnerability in a software component on the victim machine, which the attacker is aware of. The attacker prepares an attack vector, be it a special kind of file, a certain pattern in network traffic, or something else, and somehow tricks the victim to execute this attack vector.

When the victim executes the attack vector, the vulnerability is triggered. In some cases, such as in a Denial of Service (DoS) attack, this may be enough: the system might become unresponsive or exponentially slow down. But in some cases, the vulnerability may enable the attacker to inject something additional on the victim's system that will be executed.

Shellcode

Shellcode is the general term used for the small pieces of code that are commonly injected to the victim system's memory during a successful exploit. The name originates from the historical fact that the initial shellcodes used to pop up a shell interface for the attacker. Nowadays, shellcode can do many other things as well, such as install a C&C beacon or a reverse backdoor on the system. But the term shellcode has established its place in common terminology.

In an attack framework, such as Metasploit, you typically first select the initial attack vector (i.e. the "exploit"). After selecting the exploit, you choose the payload you want to execute after the successful exploit. This payload is the shellcode.

Shellcode encoders

Most vulnerabilities do not allow injecting truly arbitrary bytes to memory. To reach the exploitable code, exploit payload may need to be formatted in a specific way - for example, it can contain no NULL bytes or only ASCII characters are allowed. To work around this issue, shellcode encoders are used.

A shellcode encoder encodes the shellcode into an acceptable form, such as full ASCII. The encoders themselves often do not attempt to evade detection by security vendors, such as AV's or IPS's. Their main functionality is in modifying the code into a format which can successfully be injected into the memory. These kinds of encoders are easy to identify even with simple string matching.

There are, however, some encoders that have been created specifically to be difficult to detect by security vendors. Polymorphic encoders are especially powerful. The shellcode that a polymorphic encoder produces will be different each time the encoder is run, which makes it impossible to identify by security systems that need an initial string match to work. An excellent example of a powerful polymorphic shellcode encoder is the legendary Shikata-Ga-Nai, which is a polymorphic XOR additive feedback encoder. Shikata-Ga-Nai has a reputation as a notoriously difficult encoder to detect by modern security products.

Shellcode stagers

Sometimes, the vulnerability that is going to be exploited only permits for a very short amount of code to be injected into the memory. In this case, the attacker cannot fit the complete shellcode into the attack payload. Instead, the attacker can inject something called a stager.

A stager is a very short piece of code which will fetch a larger piece of code to be run on the victim system. This makes it possible to get a larger shellcode into the exploited system. Exploit frameworks often implement different kinds of stagers as well.

Zero-Day detection

As mentioned previously, detecting a zero-day attack is a difficult task. But, in addition to the initial attack vector which utilizes the zero-day vulnerability, an exploit also usually consists of the encoded shellcode payload. Even though the initial exploit can vary, and even though the finally executed shellcode payload can vary, one part remains: the shellcode encoder.

This means that you do not necessarily need to identify the initial attack vector. You don't even need to know what the payload would do if it got executed. All you need to do is identify the different shellcode encoders. By identifying the shellcode encoders, you have a solid defense against zero-day attacks.

Detection capabilities in the Forcepoint Next-Generation Firewall

The in-house built, DFA-based fingerprinting in the Forcepoint Next-Generation Firewall enables fast, regular expression-based fingerprinting in any network stream. Because of this, it is possible to identify even the more complex shellcode encoders from any network traffic. This is noteworthy, as many security products are only able to perform this level of detection on files - not on the entire network level.