Uber’s Secret Management Platform – Scaling Secrets Security Across Multi-Cloud

Introduction

Managing secrets—such as API keys, credentials, and tokens—across a vast, multi-cloud infrastructure is a complex challenge. Uber addressed this by developing a centralised Secret Management Platform to handle over 150,000 secrets across over 5,000 microservices. This article explores the architecture, strategies, and outcomes of Uber’s approach to secrets management.

Uber's Secret Management Platform - Scaling Secrets Security Across Multi-Cloud

Addressing Secrets Sprawl

Uber identified the risks associated with secrets scattered across codebases, configurations, and various systems. To mitigate this, they implemented both preventive and remediation strategies:

Preventive Measures: A CLI tool integrated as a pre-commit hook in Git blocks commits containing secrets, preventing them from entering code repositories.
Remediation Strategies: Real-time and scheduled scanning methods detect exposed secrets in code changes, Slack conversations, and build logs. Upon detection, the system notifies the responsible parties to revoke and remove the secrets, ensuring prompt remediation.

These measures significantly reduced the risk of secret leaks and minimised the need for incident responses.

Consolidating Secret Vaults

Before the platform’s development, Uber operated 25 secret vaults across various teams and cloud providers. This decentralised approach led to inconsistencies and management challenges. The Secrets team undertook a consolidation process:

Standardisation: Defined a Secret Management Standard to govern storage, rotation, and leak responses.
Migration: Moved secrets from custom vaults and databases into six centrally managed vaults, reducing complexity and improving oversight.

This consolidation enabled uniform security baselines and streamlined secret management across the organisation.

Building the Secret Management Platform

With centralised vaults in place, Uber developed a comprehensive Secret Management Platform featuring:

Metadata Model: Describes properties of each secret, aiding in governance and automation.
Unified Interfaces: Provides API, CLI, and web UI for seamless interaction with the platform.
Lifecycle Management: Automates secret rotation, deletion, and revocation processes.
Access Management: Implements consistent access controls across all secrets.
Integration Support: Facilitates secret exchange with third-party vendors and SaaS applications.

This platform enhanced security and improved developer productivity by simplifying secret management tasks.

Impact and Outcomes

The implementation of the Secret Management Platform led to significant improvements:

Reduced Secret Distribution: Monitoring and scoping of secret access within containers resulted in up to a 90% reduction in secrets distributed to workloads.
Enhanced Security Posture: Centralised management and standardised practices minimised the risk of secret leaks and unauthorised access.
Operational Efficiency: Consolidating vaults and automating secret lifecycle processes streamlined operations and reduced overhead.

These outcomes underscore the effectiveness of Uber’s approach to secrets management in a complex, multi-cloud environment.

Conclusion

Uber’s development of a centralised Secret Management Platform exemplifies a robust solution to the challenges of secrets management at scale. By consolidating vaults, standardising practices, and automating processes, they enhanced security, operational efficiency, and reduced risk. Organisations facing similar challenges can draw valuable insights from Uber’s strategy to strengthen their secrets management practices.