At 17:56:47 ET on 2024-12-05, a bug in our license/update server caused a large number of license emails to be sent to users with an active license. The short summary is that this was not a security incident, no customer data was exposed, no extra purchases were triggered. We don’t actually have the ability to trigger additional purchases as we don’t store payment information, our credit card processor handles those details.

If you’d like more details into the timeline, what happened to cause the bug and what we’ve done to prevent it from happening again, read on!

Background

We’ve been building out a new customer portal for several months. There are a few goals for the portal. First, we’d like to enable larger organizations to better manage their licenses, renew them in bulk, transfer between users, etc. Second, we’d like to enable users who were previous customers and have not maintained support to have access to old stable versions of Binary Ninja. Our original update server design didn’t allow users to download specific older builds and we knew that we wanted to add it at some point.

Hopefully in the next few weeks (or early 2025) we’ll be able to show the fully fledged version of this portal and it will make everyone’s life easier! (Including ours! The portal also means customers can self-service many types of transactions that currently require manual processing.)

Earlier today we pushed a change to our license server to support these changes. The change was fine during testing, right up until it… wasn’t.

Timeline

2024-12-05 13:00 ET - 16:00 ET

We migrated the license server to the new codebase and database including testing functionality such as recovering licenses, purchasing, and renewing.

17:56 ET

A user requests a license recovery, triggering the bug.

17:56 ET - 18:05 ET

Users begin to contact Vector 35 about the extra emails.

18:05 ET

Having received the many notices and confirming the duplicates, we begin investigating.

18:11 ET

Unable to identify the immediate cause, we decide to reboot the infrastructure to see if the emails stop.

18:13 ET

Confirming the reboot did not resolve the emails (in hindsight likely most were stuck in the outbound SES queue already), we powered off the newly migrated license server.

18:19 ET

We disable all outbound SES email and begin formulating customer communications.

18:29 ET

We post to our internal Slack, public Twitter, and Mastodon to let people know about the issue and actions we've taken.

19:00 ET

We continue to reply to customer communications as well as investigate all the available logs to determine the bug.

19:55 ET

After confirming that the mail was being blocked outbound, we brought the server back up so that users could still check for updates and switch versions.

20:22 ET

We investigated both the server logs and source code and identified the flaw that caused the failure. Next up, much more testing and careful monitoring!

22:25 ET

We post this blog and re-enabled email sending for license purchase/recovery!

The Bug

But wait, we said we tested license recovery. So how did a new request for a license recovery cause the flood of emails? If a user requested to recover the licenses associated with their email, and their email didn’t have a Ninja ID associated, recovery emails would be sent for ALL licenses without an associated Ninja ID. The intended logic here is to handle the case where we are migrating users off of the purely email-based license system to the new customer portal which is backed by Ninja IDs (the same ID you use to manage your Sidekick logins). In the coming weeks you’ll be able to associate your account and your license so you can also use the portal to manage your license or download previous versions after support ends!

You can see this bug in the following simplified code:

def recover():
    email = params.get('email')
    ...
    try:
        user = User.objects.get(email=email)
    except User.DoesNotExist:
        user = None
    ...
    license_selection_query = Q(email__iexact=email) | Q(user=user) # <- BUG
    licenses = License.objects.filter(license_selection_query)
    ...
    send_license_emails(licenses)

When user was None, licenses would contain all License objects where the email address matches, or has no associated user (which is all of them… oops.)

The Fix

The fix here is obviously not to match Licenses on user when user is None, or essentially changing the code to:

...
license_selection_query = Q(email__iexact=email)
if user is not None:
    license_selection_query |= Q(user=user)
...

Other Mitigations

Once we’ve recovered after a good night’s sleep we’ll re-visit this and consider other longer-term mitigations we might take.

More robust testing might have caught this condition, some rate limits on the account doing the sending at the email provider level could have prevented this, we’ll see what makes the most sense with the benefit of hindsight.

FAQ

Q: Why didn't you catch this in testing?

A: Because the license we used in testing had an associated account already and we missed the failure condition.

Q: Was any user data exposed?

A: Nope, you just got extras of the licenses you could always get at any time by going to the license recovery page.

Q: Why do the emails say I purchased Binary Ninja?!

A: Historically, our license recovery page just triggered a fresh email that looked the same as a purchase. We're changing that going forward to make it more clear which is which (sorry to introduce anxiety that we might have charged you again!)