Earlier this year, in mid-January, you might have come across this security announcement by GitHub.

In this article, I will unveil the shocking story of how I discovered CVE-2024-0200, a deceptively simple, one-liner vulnerability which I initially assessed to likely be of low impact, and how I turned it into one of the most impactful bugs in GitHub’s bug bounty history.

Spoiler: The vulnerability enabled disclosure of all environment variables of a production container on GitHub.com, including numerous access keys and secrets. Additionally, this vulnerability can be further escalated to achieve remote code execution (RCE) on GitHub Enterprise Servers (GHES), but not on GitHub.com. More on this later.

Backstory

Back in early December 2023, I was performing some research on GHES. On the day before I went on vacation, I located a potential (but likely minor) bug. Fast-forward to the day after Christmas, I finally found some time to triage and analyse this potential vulnerability. At that point, I still had zero expectations of the potential bug to be this impactful… until an accident happened.

A Quick Primer on Ruby Reflections

Before I spill the tea, allow me to begin with a brief introduction to Ruby.

Similar to JavaScript, almost everything (e.g. booleans, strings, integers) is an Object in Ruby. The Object includes the Kernel module as a mixin, rendering methods in the Kernel module accessible by every Ruby object. Notably, it is possible to do reflection (i.e. indirect method invocation) by using Kernel#send() as such:

Class HelloWorld
  def print(*args)
    puts("Hello " + args.join(' '))
  end
end

obj = HelloWorld.new()
obj.print('world') # => 'Hello World'
obj.send('print', 'world') # => 'Hello World'

As shown above, it is possible to dynamically invoke a method using Kernel#send() to perform reflection on any object. Naturally, this makes it an obvious code sink to search for, since having ability to invoke arbitrary methods on an object can be disastrous. For example, unsafe reflections with 2 controllable arguments can easily lead to arbitrary code execution:

user_input1 = 'eval'
user_input2 = 'arbitrary Ruby code here'

obj.send(user_input1, user_input2)
# is equivalent to:
obj.send('eval', 'arbitrary Ruby code here')
# which is equivalent to:
Kernel.eval('arbitrary Ruby code here')
# which has the same effect as:
eval('arbitrary Ruby code here')
# note: because everything is an object, including the current context

If you have more than 2 controllable arguments, then it is also trivial to achieve arbitrary code execution by calling send() repeatedly:

obj.send('send', 'send', 'send', 'send', 'eval', '1+1')
# will call: 
obj.send('send', 'send', 'send', 'eval', '1+1')
# ...
obj.send('eval', '1+1')
# to finally call:
eval('1+1')

You can read more about unsafe reflections in Phrack Issue 0x45 by @joernchen or this Ruby security discussion published on Seebug (in Chinese).

Interestingly enough, I did not come across any discussions on unsafe reflections with only 1 controllable argument in Kernel#send() as shown below:

user_input = 'method_name_here'
obj.send(user_input)

At first glance, it seems rather difficult to escalate impact at all in this scenario. From the list of default methods inherited from Object, I identified the following useful methods:

# Disclosing filepaths:
obj.send('__dir__') # leak resolved absolute path to directory containing current file
obj.send('caller')  # return execution call stack, and may leak filepaths

# Disclosing class name
obj.send('class')

# Disclosing method names
obj.send('__callee__')
obj.send('__method__')
obj.send('matching_methods')
obj.send('methods') # Object#methods() returns list of public and protected methods
obj.send('private_methods')
obj.send('protected_methods')
obj.send('public_methods')
obj.send('singleton_methods')

# Disclosing variable names
obj.send('instance_variables')
obj.send('global_variables')
obj.send('local_variables')

# Stringify variable
obj.send('inspect') # calls to_s recursively
obj.send('to_s')    # string representation of the object

# Read from standard input
obj.send('gets')
obj.send('readline')
obj.send('readlines')

# Terminates process (please exercise caution)
obj.send('abort')
obj.send('fail')
obj.send('exit')
obj.send('exit!')

These methods may come in handy when attempting to gather more information on the target, especially when performing blind, unsafe reflections.

However, in the case of GitHub, this won’t be necessary since we can audit the source code of GHES, which is largely identical to the one deployed on GitHub.com. Now, we are ready to move on to discuss the vulnerability.

Discovering the Vulnerability

Note: The source code presented below were extracted from GitHub Enterprise Server (GHES) 3.11.0 to pinpoint the root cause of the vulnerability.

Doing a quick search on the codebase, I found an unvalidated Kernel#send() call in Organizations::Settings::RepositoryItemsComponent found in app/components/organizations/settings/repository_items_component.rb:

...
class Organizations::Settings::RepositoryItemsComponent < ApplicationComponent
  def initialize(organization:, repositories:, selected_repositories:, current_page:, total_count:, data_url:, aria_id_prefix:, repository_identifier_key: :global_relay_id, form_id: nil)
    @organization = organization
    @repositories = repositories
    @selected_repositories = selected_repositories
    @show_next_page = current_page * Orgs::RepositoryItemsHelper::PER_PAGE < total_count
    @data_url = data_url
    @current_page = current_page
    @aria_id_prefix = aria_id_prefix
    @repository_identifier_key = repository_identifier_key # [2]
    @form_id = form_id
  end
  ...
  def identifier_for(repository)
    repository.send(@repository_identifier_key) # [1]
  end
  ...
end

At [1], repository.send(@repository_identifier_key) is invoked in the identifier_for() method without any prior input validation on @repository_identifier_key (set at [2]). This allows all methods accessible by the object (including private or protected methods, and any other methods inherited from ancestor classes) to be invoked.

The identifier_for() method of the Organizations::Settings::RepositoryItemsComponent class is used in app/components/organizations/settings/repository_items_component.html.erb (the template file to be rendered and returned within the HTTP response body) at [3]:

<%# erblint:counter ButtonComponentMigrationCounter 1 %>
<% @repositories.each do |repository| %>
  <li <% unless first_page? %> hidden <% end %> class="css-truncate d-flex flex-items-center width-full">
    <input
      <%= "form=#{@form_id}" if @form_id.present? %>
      type="checkbox" name="repository_ids[]"
      value="<%= identifier_for(repository) %>" # [3]
      id="<%= @aria_id_prefix %>-<%= repository.id %>"
...

Backtracing further, it can be seen that Organizations::Settings::RepositoryItemsComponent objects are initialised in app/controllers/orgs/actions_settings/repository_items_controller.rb:

class Orgs::ActionsSettings::RepositoryItemsController < Orgs::Controller
  ...

  def index
    ...
    respond_to do |format|
      format.html do
        render(Organizations::Settings::RepositoryItemsComponent.new(
          organization: current_organization,
          repositories: additional_repositories(selected_repository_ids),
          selected_repositories: [],
          current_page: page,
          total_count: current_organization.repositories.size,
          data_url: data_url,
          aria_id_prefix: aria_id_prefix,
          repository_identifier_key: repository_identifier_key, # [4]
          form_id: form_id
        ), layout: false)
      end
    end
  end

  ...

  def rid_key
    params[:rid_key] # [6]
  end

  ...

  def repository_identifier_key
    return :global_relay_id unless rid_key.present? # [5]
    rid_key
  end

  ...
end

At [4], the result of the repository_identifier_key() method is passed as the repository_identifier_key keyword argument when initialising the Organizations::Settings::RepositoryItemsComponent object. At [5], in repository_identifier_key(), observe that :global_relay_id is returned only when the return value of rid_key() is absent. Otherwise, the repository_identifier_key() method simply passes on the return value from rid_key() – params[:rid_key] (at [6]).

Putting it all together, the unsafe reflection repository.send(@repository_identifier_key) allows for a “zero-argument arbitrary method invocation” on a Repository object.

In Search of Impact

This is exactly the same scenario I discussed earlier. Unfortunately, none of the options I shared earlier are applicable in this case – the information is likely available to us already, or they do not do anything useful for us at this point. So, how can we escalate the impact further?

It is crucial to recognise that we are not limited to methods inherited from Object – we can expand the search of candidate methods by looking at methods accessible by a Repository object.

Next, let’s refer back to the assumption of having a “zero-argument arbitrary method invocation”. What does that even mean? Can we only invoke methods accepting no arguments at all?

The answer is: No. Surprise!

Actually, this is a common misassumption with a pretty straightforward counter-example to disprove it (as shown below):

class Test
  # zero arguments required
  def zero_arg()
  end

  # 1 positional argument required
  def one_pos_arg(arg1)
  end

  # 2 positional arguments required, but second argument has default value
  def one_pos_arg_one_default_pos_arg(arg1, arg2 = 'default')
  end

  # 2 positionals argument required, but both arguments have default values
  def two_default_pos_args(arg1 = 'default', arg2 = 'default')
  end

  # 1 keyword argument (similar to positional argument) required (no default value)
  def one_keyword_arg(keyword_arg1:)
  end

  # 1 keyword argument (has default value) required
  def one_default_keyword_arg(keyword_arg1: 'default')
  end

  # 1 positional (no default value) & 1 keyword argument (has default value) required
  def one_pos_arg_one_default_keyword_arg(arg1, keyword_arg2: 'default') 
  end
end

obj = Test.new()
obj.send('zero_arg')                            # => OK
obj.send('one_pos_arg')                         # => in `b': wrong number of arguments (given 1, expected 0) (ArgumentError)
obj.send('one_pos_arg_one_default_pos_arg')     # => in `c': wrong number of arguments (given 1, expected 1..2) (ArgumentError)
obj.send('two_default_pos_args')                # => OK
obj.send('one_keyword_arg')                     # => in `e`: missing keyword: :keyword_arg1 (ArgumentError)
obj.send('one_default_keyword_arg')             # => OK
obj.send('one_pos_arg_one_default_keyword_arg') # => OK

Clearly, we are able to invoke methods requiring arguments just fine – so long as they have default values assigned to them!

With these two tricks in our bag, we are now ready to start searching for candidate methods… but how? We can simply grep until we find something useful, but this will be a tedious process. The main Docker image containing the Ruby on Rails application source code contains more than a whopping 100k files (~1.5 GB), so we clearly need a better strategy.

A simple solution to this complex task is just to drop into a Rails console in a test GHES setup, and use reflection to aid us in our quest:

repo = Repository.find(1) # get first repo
methods = [ # get names of all methods accessible by Repository object
  repo.public_methods(),
  repo.private_methods(),
  repo.protected_methods(),
].flatten()

methods.length() # => 5542

Yes, you read that correctly. I was quite shocked when I saw the output too.
Why on earth does the Repository object even have 5542 methods? Well, to be fair, most of it came from autogenerated code by Ruby on Rails, which leverages Ruby metaprogramming to define getters/setter methods on objects.

Let’s further reduce the search space by finding methods matching the criteria (i.e. no required positional or keyword arguments that do not have default values). This is because we need to prevent Ruby from throwing ArgumentError due to obvious mismatch of the number of required arguments. Going back to the previous example on the Test class, let’s examine the arity of the method:

class Test
  # zero arguments required
  def zero_arg()
  end

  # 1 positional argument required
  def one_pos_arg(arg1)
  end

  # 2 positional arguments required, but second argument has default value
  def one_pos_arg_one_default_pos_arg(arg1, arg2 = 'default')
  end

  # 2 positionals argument required, but both arguments have default values
  def two_default_pos_args(arg1 = 'default', arg2 = 'default')
  end

  # 1 keyword argument (similar to positional argument) required (no default value)
  def one_keyword_arg(keyword_arg1:)
  end

  # 1 keyword argument (has default value) required
  def one_default_keyword_arg(keyword_arg1: 'default')
  end

  # 1 positional (no default value) & 1 keyword argument (has default value) required
  def one_pos_arg_one_default_keyword_arg(arg1, keyword_arg2: 'default') 
  end
end

obj = Test.new()
obj.method('zero_arg').arity()                            # => 0
obj.method('one_pos_arg').arity()                         # => 1
obj.method('one_pos_arg_one_default_pos_arg').arity()     # => -2
obj.method('two_default_pos_args').arity()                # => -1
obj.method('one_keyword_arg').arity()                     # => 1
obj.method('one_default_keyword_arg').arity()             # => -1
obj.method('one_pos_arg_one_default_keyword_arg').arity() # => -2

It appears that only methods with arity of 0 or -1 can be used by us in this case.

Now, we can filter the list of candidate methods further:

repo = Repository.find(1)  # get first repo
repo_methods = [           # get names of all methods accessible by Repository object
  repo.public_methods(),
  repo.private_methods(),
  repo.protected_methods(),
].flatten()

repo_methods.length()      # => 5542
candidate_methods = repo_methods.select() do |method_name|
  [0, -1].include?(repo.method(method_name).arity())
end
candidate_methods.length() # => 3595

I guess that is slightly better…? Metaprogramming can be a curse sometimes. 😅

While I could further reduce the search space, I didn’t want to risk having missing out on any potentially useful functions. It is probably a good idea to scan through the output to get a better sensing of what methods are available first before further processing.

Let’s dump the location of where the methods are defined:

candidate_methods.map!() do |method_name|
  method = repo.method(method_name)
  [
    method_name,
    method.arity(),
    method.source_location()
  ]
end
puts(candidate_methods.sort())

The output is a long list of 3595 methods and their location:

[
  [:!, [0, nil]],
  [:Nn_, [-1, ["/github/vendor/gems/3.2.2/ruby/3.2.0/gems/fast_gettext-2.2.0/lib/fast_gettext/translation.rb", 65]]],
  [:_, [-1, ["/github/vendor/gems/3.2.2/ruby/3.2.0/gems/gettext_i18n_rails-1.8.1/lib/gettext_i18n_rails/html_safe_translations.rb", 10]]],
  [:__callbacks, [0, ["/github/vendor/gems/3.2.2/ruby/3.2.0/gems/activesupport-7.1.0.alpha.bb4dbd14f8/lib/active_support/callbacks.rb", 70]]],
  ...
  [:xcode_clone_url, [-1, ["/github/app/helpers/url_helper.rb", 218]]],
  [:xcode_project?, [0, ["/github/packages/repositories/app/models/repository/git_dependency.rb", 323]]],
  [:xcode_urls_enabled?, [0, ["/github/app/helpers/url_helper.rb", 213]]],
  [:yield_self, [0, ["<internal:kernel>", 144]]]
]

Triaging Candidate Methods

I started triaging the list of potentially useful methods, but as I was testing locally, I realised my test GHES installation wasn’t working correctly somehow and had to be re-installed. At this point, I noticed most of the methods would likely only affect my own organisation repositories, or allow me to leak some information like file paths on the server, which may come in handy in the future.

I didn’t really want to waste precious time doing nothing while waiting for the re-installation to complete, so I decided to test it on the production GitHub.com server in my own test organisation given the current potential impact achievable.

Nothing could possibly go wrong, right…? 🙈

Wrong. I was completely off the mark. The following response came back shortly after I began testing on the production GitHub.com server, leaving me completely speechless and stunned in disbelief:

That’s ~2MB worth of environment variables belonging to GitHub.com containing a massive list of access keys and secrets within the response body. How did these secrets end up here?!

Getting the Environment Variables

Let’s examine the Repository::GitDependency module (at packages/repositories/app/models/repository/git_dependency.rb) included by the Repository containing the dangerous method nw_fsck():

module Repository::GitDependency
  ...
  def nw_fsck(trust_synced: false)
    rpc.nw_fsck(trust_synced: trust_synced)
  end
  ...
end

Note: In Ruby, the last evaluated line of any method/block is the implicit return value.

This nw_fsck() method is extremely inconspicuous, but holds a wealth of information. To understand why, let’s examine the GitRPC backend implementation in vendor/gitrpc/lib/gitrpc/backend/nw.rb:

module GitRPC
  class Backend
    ...
    rpc_writer :nw_fsck, output_varies: true
    def nw_fsck(trust_synced: false)
      argv = []
      argv << "--connectivity-only"
      argv << "--trust-synced" if trust_synced
      spawn_git("nw-fsck", argv) # [7]
    end
    ...
  end
end

At [7], the return value of the nw_fsck() method is the git process created using the spawn_git() method. The spawn_git() method eventually calls and returns GitRPC::Native#spawn():

...
module GitRPC
  ...
  class Native
    ...
    def spawn(argv, input = nil, env = {}, options = {})
      ...
      {
        # Report unhandled signals as failure
        :ok        => !!process.status.success?,
        # Report the exit status as the signal number if we have it
        :status    => process.status.exitstatus || process.status.termsig,
        :signaled  => process.status.signaled?,
        :pid       => process.status.pid,
        :out       => process.out,
        :err       => process.err,
        :argv      => argv,
        :env       => env, # [8]
        :path      => @path,
        :options   => options,
        :truncated => truncated,
      }
    end
    ...
  end
end

Observe that the value returned by GitRPC::Native.spawn() is a Hash object containing the environment variables passed to the git process at [8]. Digging deeper, the list of environment variables passed to the git process created can be found in vendor/gitrpc/lib/gitrpc/backend.rb:

...
module GitRPC
  ...
  class Backend
    ...
    def self.environment
      @environment ||= ENV.to_h.freeze # [10]
    end
    ...
    def native
      @native ||= GitRPC::Native.new(path, native_env, native_options)
    end
    ...
    def native_env
      env = GitRPC::Backend.environment.dup # [9]
      env.merge!(options[:env] || {})
      env.merge!(GitRPC.extra_native_env || {})
      env["GITHUB_TELEMETRY_LOGS_NOOP"] = "true"
      env["GIT_DIR"] = path
      env["GIT_LITERAL_PATHSPECS"] = "1"
      env["GIT_SOCKSTAT_VAR_via"] = "gitrpc"
      if options[:info]
        git_sockstat_var_options.each do |(prefix, sym)|
          env["GIT_SOCKSTAT_VAR_#{sym}"] = "#{prefix}#{options[:info][sym]}" if options[:info][sym]
        end
      end
      if alternates = alternate_object_paths
        env["GIT_ALTERNATE_OBJECT_DIRECTORIES"] = alternates.join(":")
      end
      env
    end
    ...
  end
end

At [9], GitRPC::Backend#native_env() duplicates the environment variable Hash object returned by GitRPC::Backend::environment() at [10], which is basically a copy of all environment variables passed to Rails.

Since GitRPC::Native#spawn() returns the list of environment variables in a Hash, and Repository::GitDependency#nw_fsck() kindly returns the Hash to us, we are able to disclose all the environment variables passed to Rails!

The Actual Impact

I inadvertently gained access to a total of 1220 environment variables (~2MB) containing a lot production access keys. I immediately stopped my testing, reached out to folks at GitHub to alerting them of this incident and proceeded to submit a vulnerability report soon after.

Special thanks to Simon Gerst and Jorge Rosillo for their help in getting in contact with someone from the GitHub Security Incident Response Team (SIRT) team so that I could give them an early heads-up notice on this.

Getting Remote Code Execution (RCE)

The following day, I continued further research to see if it is possible to escalate the impact further and achieve remote code execution. Of course, not with the access keys! I didn’t want to mess with the production GitHub.com environment further, so I continued my research using my new test GHES setup.

I quickly noticed that the list of environment variables included ENTERPRISE_SESSION_SECRET, which is used for signing marshalled data stored in session cookies:

As previously demonstrated in @iblue’s remote code execution in GitHub Enterprise management console, having knowledge of ENTERPRISE_SESSION_SECRET value allows an attacker to sign arbitrary serialised data.

Ruby on Rails implements session storage using a cryptographically signed serialized Ruby Hash. This Hash is serialized into a cookie using Marshal.dump and subsequently deserialized using Marshal.load. If an attacker can construct a valid signature, they can create a session cookie that contains arbitrary input passed to Marshal.load. As noted by the Ruby documentation for Marshal.load, this can result in code execution:

By design, ::load can deserialize almost any class loaded into the Ruby process. In many cases this can lead to remote code execution if the Marshal data is loaded from an untrusted source.

As a result, ::load is not suitable as a general purpose serialization format and you should never unmarshal user supplied input or other untrusted data.

This is a clear path to obtaining RCE, but there are still a few more hurdles to resolve (which I will discuss another time). However, this environment variable is not set on GitHub.com, so there’s no quick and easy way to get remote code execution on GitHub.com except to use the access keys (but don’t do this, please!).

To reiterate, this vulnerability can be chained to achieve RCE on GHES, but GitHub.com is not affected.

Exploit Conditions

This vulnerability affects GitHub.com and any GitHub Enterprise Server with GitHub Actions enabled. An attacker also needs to have the organisation owner role.

Suggested Mitigations

1. Validate the rid_key parameter in the Orgs::ActionsSettings::RepositoryItemsController class against an allowlist to ensure that only intended methods of the Repository class can be invoked.
2. Revoke and regenerate all secrets used in GitHub.com / any GitHub Enterprise Server that may have been compromised.
3. Consider spawning git processes with a minimal set of environment variables required for functioning instead. Currently, a total of 1220 environment variables is being passed to the git process on GitHub.com.

Detection Guidance

It is possible to detect the exploitation of this vulnerability by checking the server’s access logs for all requests made to /orgainzations/<organisation_name>/settings/actions/repository_items with an abnormal rid_key parameter value set. However, it is worthwhile to note that Rails accepts rid_key parameter supplied within the request body as well.

Timeline

• 2023-12-26 Vendor Disclosure
• 2023-12-26 Initial Vendor Contact
• 2023-12-26 Hotfixed on Github.com Production Servers
• 2023-12-28 Sent RCE Report to Vendor
• 2024-01-16 Vendor Patch & Announcement Release
• 2024-05-06 Public Release

Closing Thoughts

Regrettably, this vulnerability was discovered and exploited at a really inopportune time (day after Christmas). I want to express my sincere apologies and gratitude to all Hubbers involved and working during the Christmas/New Year festive period, since the amount of work created for the Hubbers as a consequence of this bug report must have been insanely huge. It is, however, incredibly impressive to see them get this vulnerability patched quickly, rotate all secrets (an awfully painful process), as well as running and concluding a full investigation to confirm that this vulnerability had not been exploited in-the-wild previously.

I hope you have enjoyed reading the whole process on how I managed to exploit this inconspicuous, unsafe reflection bug with limited control and turning it one of the most impactful bugs on GitHub. Thanks for reading!