It’s the weekend and I was writing malware for fun. A thought popped into my mind: what are the other ways we can implement RC4 in our code to encrypt a payload? So here is a short article on it.

The classic RC4 implementation is fairly long. It usually includes key scheduling, permutation arrays, and additional logic that increases code size. It works, but it is not always ideal when you want something compact for payload encryption. While searching for alternative approaches, I came across an undocumented Windows API called SystemFunction032, exported by Advapi32.dll, which already implements RC4 internally.

This means we can use the Windows API directly instead of implementing RC4 ourselves.

What is SystemFunction032

SystemFunction032 is an undocumented Windows function that performs RC4 encryption directly on a buffer. Since RC4 is symmetric, the same function is used for both encryption and decryption.

The function expects two USTRING structures:

• One describing the data buffer
• One describing the key buffer

The data buffer is modified in place by the function.

The USTRING Structure

typedef struct {
    DWORD Length;
    DWORD MaximumLength;
    PVOID Buffer;
} USTRING;

This structure describes a buffer in memory:

• Buffer points to the data
• Length specifies how much data to process
• MaximumLength defines the buffer capacity

Function Prototype

We dynamically resolve the function from Advapi32.dll:

typedef NTSTATUS (WINAPI* fnSystemFunction032)(
    USTRING* Data,
    USTRING* Key
);

The encryption happens in place, meaning the original buffer gets modified directly.

RC4 Helper Using SystemFunction032

#include <windows.h>
#include <stdio.h>
typedef struct {
    DWORD Length;
    DWORD MaximumLength;
    PVOID Buffer;
} USTRING;
typedef NTSTATUS (WINAPI* fnSystemFunction032)(
    USTRING* Data,
    USTRING* Key
);
BOOL Rc4EncryptionViaSystemFunc032(
    PBYTE pRc4Key,
    PBYTE pPayloadData,
    DWORD sRc4KeySize,
    DWORD sPayloadSize
) {
    USTRING Data = { sPayloadSize, sPayloadSize, pPayloadData };
    USTRING Key  = { sRc4KeySize, sRc4KeySize, pRc4Key };
    fnSystemFunction032 SystemFunction032 =
        (fnSystemFunction032)GetProcAddress(
            LoadLibraryA("Advapi32.dll"),
            "SystemFunction032"
        );
    return SystemFunction032(&Data, &Key) == 0;
}

Using It to Encrypt a Payload

int main() {
BYTE data[] = "HelloWorld";
    BYTE key[]  = "secretkey";
    DWORD dataSize = sizeof(data) - 1;
    DWORD keySize  = sizeof(key) - 1;
    printf("Original: %s\n", data);
    Rc4EncryptionViaSystemFunc032(key, data, keySize, dataSize);
    printf("Encrypted: ");
    for (int i = 0; i < dataSize; i++)
        printf("%02X ", data[i]);
    printf("\n");
    Rc4EncryptionViaSystemFunc032(key, data, keySize, dataSize);
    printf("Decrypted: %s\n", data);
    return 0;
}

Example output:

Original: HelloWorld
Encrypted: A1 3F 9C 22 11 7B 8E 44 29 90
Decrypted: HelloWorld

Why This Works

RC4 is a symmetric stream cipher. Encryption and decryption are identical operations. Calling the function twice with the same key restores the original buffer.

Use Cases for Payload Encryption

This approach is useful for:

• shellcode encryption
• payload obfuscation
• runtime decryption
• encrypted configuration blobs
• loader development

Typical workflow:

1. Encrypt payload
2. Store encrypted bytes
3. Decrypt at runtime
4. Execute payload

Final Note

Using SystemFunction032 avoids implementing RC4 manually and keeps the code compact. The API already handles the algorithm internally, and we only need to wrap our buffers using USTRING and call the function. This makes it a clean and practical approach for payload encryption.

That’s it for this short article. Enjoy yourself.