LiteLLM's Own Post-Mortem: a Master Key Left Live for 136 Days

The root cause: a key committed in May and never revoked

LiteLLM has published the full root-cause analysis of the October 21 supply-chain attack, and the starting point is almost embarrassingly small. On May 27, an opencode.json file "inadvertently contained the master key to a deployment of the LiteLLM Agent Platform" and was committed to the public LiteLLM-Labs/lite-harness repository at 18:53:14 UTC. The commit was force-pushed away and the exposed keys were flagged for revocation, but the post-mortem states plainly that "key revocation did not occur for the master key." From May 27 to the attacker's first use on October 10 is 136 days.

The report, published on October 23 by the company's director of security and an AI engineer, is described as the canonical source of truth, superseding earlier status updates. It confirms the shape of the attack this blog covered when the backdoored v1.107.0 wheels were pulled from PyPI, and fills in the part that mattered most: how the attacker got in.

The attack chain, hour by hour

On October 10 at 21:27 UTC, an attacker used the master key to list agent configurations, and one of them contained "a GitHub CLI OAuth token associated with our GitHub organization owner's account", granting owner-level access to all repositories. The report notes GitHub CLI tokens are issued after two-factor authentication and are "not subject to GitHub's 'super-user' 2fa reauth flows".

Over the following days the attacker ran throwaway CI branches whose workflows posted secrets to webhook endpoints, harvesting a second and a third maintainer's tokens, then deleted the workflow run records to hide the exfiltration. On October 12 came further exfiltration from private repositories, targeting CI secrets and inference API provider keys.

The release hijack on October 21 was a chain of small failures. Branch protection rejected the attacker's first push; one second later the attacker disabled branch protections and force-pushed with the owner's token. The staging workflow was pointed at a lookalike organization, Berri-AI, at a commit that "exists only on the attacker's fork". The production approval gate, normally requiring "a special account locked to a physical device in our office", had been relaxed days earlier to allow any two engineers, and the second maintainer's stolen token approved it. The report observes the attacker could very well have relaxed the restriction themselves but used separate tokens to evade audit-log detection. Seven backdoored wheels reached PyPI at 11:39 UTC, followed by Docker Hub and GHCR images, and afterward the third stolen token deleted the release run records and their logs. Aviatrix's independent breakdown of the same chain matches the RCA step for step.

Detection in 20 minutes, cover-up in minutes too

The contamination was caught fast: an issue flagging the malicious wheels was opened within 20 minutes of publication, PyPI quarantined the release about two hours in, and the Docker tags were pulled or repointed to clean images, a sequence Datadog's write-up also reconstructs from release telemetry. The malicious payload, per the RCA, added a certificate authority to the Python trust store, with deeper payload analysis deferred to the incident report and Socket's write-up.

The response posture is the part other maintainers should copy. LiteLLM removed nearly all organization members, locked its registries, and assumed "every possible token, key, or authenticator belonging to our organization was exfiltrated", revoking tokens across internal systems and vendors rather than trying to reconstruct exactly what was taken. A clean release removing the planted certificate authority shipped the same week, with process improvements promised to follow.

What the RCA actually teaches

Three lessons sit above the rest. A secret committed to a public repository must be treated as burned even if the commit disappears, because force-pushing removes the commit but not the copies, caches, or alerts that may have captured it. Second-factor rules that have been relaxed for convenience are invisible attack surface, and the attacker in this case specifically avoided using the most powerful token in the chain so audit logs would look normal. And approval gates are only as strong as the tokens that can satisfy them: three stolen maintainer tokens turned a two-person approval into a formality.

For teams running LiteLLM or any proxy in front of production models, the practical checklist follows directly from the report: rotate anything that was ever near the incident, scope release tokens to service accounts, keep device-bound approvals unrelaxed, and audit fork-based release sources and branch-protection changes as first-class alerts. The wheels were live for roughly two hours; the master key was live for 136 days. The second number did far more damage.