When CVE-2024-21762 and CVE-2024-23113 were patched in February 2024, Bishop Fox analyzed the patches to better understand the technical details of the vulnerabilities and developed a CVE-2024-21762 vulnerability scanner. While embarking on our analysis, we noticed that Fortinet recently added another layer of encryption to their firmware format.

In this blog post, we examine how the new encryption scheme works and provide a tool to decrypt the root filesystem for x86-based FortiOS images.

Note: Optistream and Fox-IT have each independently published similar research and tooling related to FortiGate firmware analysis.

What Changed?

Fortinet implements the vast majority of functionality in the monolithic multicall /bin/init binary. On old versions, extracting this file involved a few steps:

1. Decrypt the update file with FortiCrack
2. Mount the update file to obtain rootfs.gz
3. Extract rootfs.gz to obtain bin.tar.xz
4. Extract /bin/init from bin.tar.xz

On new versions, extraction starts out the same, however; we run into issues at step 3:

$ gunzip rootfs.gz
 
gzip: rootfs.gz: not in gzip format 
$ file rootfs.gz 
rootfs.gz: data 
$ xxd rootfs.gz|head 
00000000: 062d 02db eb25 04a0 b529 65d6 b9bf e616  .-...%...)e..... 
00000010: 5180 ddb2 4024 bec8 feb7 2ba7 d52b b5e0  Q...@$....+..+.. 
00000020: fd39 53eb 6bd0 02df e53e 2b47 b241 3233  .9S.k....>+G.A23 
00000030: da83 30a6 c8bf f214 dfc1 5abd 8729 2886  ..0.......Z..)(. 
00000040: 48da 4981 43b9 dc54 3edc 19ac dac7 6aab  H.I.C..T>.....j.

The file command fails to identify the filetype of rootfs.gz. Upon examining a hexdump, there is no obvious file header, and the data appears to be very high entropy. This suggests that the file is encrypted.

Finding the Decryption Code

Since rootfs.gz is the initramfs for the system, we know that the decryption must occur in the kernel or the bootloader. Since the update image does not include a new bootloader, we begin our search in the kernel.

In the update file, the kernel image is called flatkc. This file can be converted to an ELF binary using vmlinux-to-elf. The ELF binary can then be loaded into the disassembler or decompiler of your choice for analysis. vmlinux-to-elf uses the kernel symbol table to generate ELF symbols, which means we have a lot of readable symbol names. We quickly found a handful of symbols with names like fgt_verify_* and fgt_verifier_*. These symbols include fgt_verify_decrypt, which is called by fgt_verify_initrd and which calls fgt_verifier_key_iv.

fgt_verify_decrypt calls fgt_verifier_key_iv to generate a key and IV, then passes those values to crypto_chacha20_init and decrypts a region of memory by calling chacha20_docrypt.

fgt_verifier_key_iv retrieves hardcoded data from a global buffer and uses the SHA256 algorithm to derive the key and IV. We compared a handful of images and found that every update file uses a different random value for this global buffer data.

As a side note, this code is all compiled into the Linux kernel and therefore should be covered by the GPL. We have yet to obtain GPL source code for any components of FortiOS or the U-Boot based FortiBootloader, but we would be interested in seeing Fortinet’s other modifications to these open-source components if any readers have them.

Key Derivation and Decryption

Based on the code, we made a small script based on Objdump, LIEF, and PyCryptodome which extracts the random data, calculates the key, and decrypts the rootfs image.

This code works by finding the four calls to sha256_update in the disassembly, and then finding the preceding mov instructions which load the data and size arguments (rsi and rdx registers, respectively). From there, LIEF is used to extract the data from the ELF using the virtual address.

Once we have the key and IV, we still need to decrypt the file. It turns out that Fortinet’s usage of ChaCha20 is not compliant with RFC 7539. Specifically, the RFC requires a 12-byte IV and keeps an internal 12-byte counter to fill the remaining 4 bytes of state, and Fortinet treats the counter and state as a single 16-byte value. We can work around this using the seek() function in PyCryptodome. We treat the first four bytes of the IV as a little-endian integer, multiply it by 64 (the block size of the cipher), and pass that value as the argument to seek().

Putting that all together, we’re left with the following script:

from hashlib import sha256
from lief import ELF
from Crypto.Cipher import ChaCha20
import subprocess, sys

if len(sys.argv)!=4:
    print("Usage: {} <flatkc.elf> <rootfs.gz> <rootfs_decrpyted.gz>".format(sys.argv[0]))
    exit()
filename=sys.argv[1]
rootfs_in=sys.argv[2]
rootfs_out=sys.argv[3]

e=ELF.parse(filename)

cmd="objdump -Mintel --disassemble=fgt_verifier_key_iv {} |grep -B3 '_update'|grep -v rdi|grep -v call|grep :|cut -d'\t' -f3".format(filename)
lines=subprocess.check_output(cmd, shell=True).decode().split("\n")
groups=zip(*[iter(lines)]*2)
data=[]
for i in groups:
    vals={}
    for j in i:
        if "edx" in j:
            vals["edx"]=int(j.split(",")[-1],16)
        elif "rsi" in j:
            vals["rsi"]=int(j.split(",")[-1],16)
        else:
            print("Unexpected instruction!")
            exit(1)
    if "edx" in vals and "rsi" in vals:
        data.append(vals)
    else:
        print("Failed to find instructions")
        exit(2)

assert len(data)==4, "failed to find all values"
sha=sha256()
sha.update(bytes(e.get_content_from_virtual_address(data[0]['rsi'], data[0]['edx'])))
sha.update(bytes(e.get_content_from_virtual_address(data[1]['rsi'], data[1]['edx'])))
key=sha.digest()
sha=sha256()
sha.update(bytes(e.get_content_from_virtual_address(data[2]['rsi'], data[2]['edx'])))
sha.update(bytes(e.get_content_from_virtual_address(data[3]['rsi'], data[3]['edx'])))
iv=sha.digest()[:16]

print("Key: "+key.hex())
print("IV: "+iv.hex())

chacha=ChaCha20.new(key=key, nonce=iv[4:])
counter=int.from_bytes(iv[:4],'little')
chacha.seek(counter*64)

ciphertext=open(rootfs_in,"rb").read()
plaintext=chacha.decrypt(ciphertext)
open(rootfs_out,"wb").write(plaintext)

And we can decrypt an image as follows:

$ vmlinux-to-elf flatkc flatkc.elf
[+] Kernel successfully decompressed in-memory (the offsets that follow will be given relative to the decompressed binary)
[+] Version string: Linux version 3.2.16 (root@build) (gcc version 10.3.0 (GCC) ) #2 SMP Wed Nov 1 23:30:18 UTC 2023
[+] Guessed architecture: x86_64 successfully in 0.72 seconds
[+] Found kallsyms_token_table at file offset 0x0063fcf8
[+] Found kallsyms_token_index at file offset 0x00640040
[+] Found kallsyms_markers at file offset 0x0063fa88
[+] Found kallsyms_names at file offset 0x00607bb0
[+] Found kallsyms_num_syms at file offset 0x00607ba8
[i] Null addresses overall: 0.0100644 %
[+] Found kallsyms_addresses at file offset 0x005e0ea8
[+] Successfully wrote the new ELF kernel to flatkc.elf
$ python3 forti_decrypt.py flatkc.elf rootfs.gz rootfs_decrypted.gz
Key: b76459e38472620ca3814bbf9b5fbb4b71473a1debd9fe40b3581990ef67e9a8
IV: 929e14825692c44d54aa5e5195a4c8d1
$ file rootfs.gz rootfs_decrypted.gz
rootfs.gz:           data
rootfs_decrypted.gz: gzip compressed data, last modified: Tue Dec 19 01:14:30 2023, from Unix, original size modulo 2^32 1435315669 gzip compressed data, unknown method, ASCII, extra field, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 1435315669

Conclusion

Accessing the decrypted code running on a system is critical to understanding the technical details of many major vulnerabilities. With access to decrypted firmware, our team has developed tools for detection, fingerprinting, and vulnerability testing. This value is directly passed on to our customers in the form of security insights and greater visibility into their attack surface. Obfuscating firmware will never prevent us (or attackers) from understanding it, but it does delay the vulnerability research that plays a key part in keeping the internet safe and secure.