LiteLLM PyPI Supply Chain Attack: What Happened & How to Fix It

Overview

On March 24, 2026, two malicious versions of the litellm Python package - v1.82.7 and v1.82.8 - were published to PyPI as part of a broader supply chain campaign attributed to a threat actor group known as TeamPCP. The packages were live for approximately 40 minutes before being quarantined by PyPI, during which they accumulated tens of thousands of downloads.

LiteLLM is a widely used open-source library that provides a unified interface for calling over 100 large language model (LLM) provider APIs, including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, and Azure OpenAI. It is commonly used in AI agent frameworks, MCP servers, CI/CD pipelines, and production inference infrastructure. Because it sits between applications and multiple AI service providers, it frequently has access to API keys and cloud credentials - making it a high-value target.

This post covers the attack background, the payload behavior, how security teams can detect it using network and endpoint telemetry, indicators of compromise, and recommended remediation steps.

Action Required: Versions 1.82.7 and 1.82.8 of litellm python package are confirmed malicious. Both have been removed from PyPI but may still be present in cached environments or pinned CI/CD pipelines.

Background: How This Happened

The LiteLLM compromise is the third incident in a campaign attributed to TeamPCP, a threat actor group that has been targeting open-source tooling used in developer and security workflows.

The known campaign timeline:

• The threat actor group TeamPCP compromised the Trivy security scanner as part of a broader CI/CD supply chain attack.
• LiteLLM’s CI/CD pipeline, which depended on Trivy, was indirectly affected.
• During this window, PyPI publishing credentials were exposed, allowing attackers to gain unauthorized access to the release pipeline.

• 10:39 AM -  Attackers used the stolen credentials to publish LiteLLM v1.82.7 to PyPI
  Contained a malicious payload embedded in proxy_server.py
• 10:52 AM - A second version, v1.82.8, was released
  Introduced a more aggressive persistence mechanism via a .pth file (litellm_init.pth)
  This ensured automatic execution on Python interpreter startup

• 11:48 AM - Callum McMahon reported suspicious behavior
  • Opened GitHub Issue #24512
  • Triggered by a system crash caused by a fork bomb-like effect from the malicious .pth file

Phase 4: Attacker Interference

• 12:44 PM - The attacker, still having maintainer-level access:

  • Closed the GitHub issue
  • Flooded the thread with bot-generated noise
  • Attempted to delay or suppress investigation

Phase 5: Containment & Response

• 1:38 PM - PyPI administrators intervened:

  • Quarantined the entire LiteLLM package

  • Blocked new downloads

  • Removed the malicious versions from distribution
    

LiteLLM's own security advisory states: 'We believe that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow.' The incident illustrates a cascading trust failure: a compromised upstream tool in one CI/CD pipeline handed attackers the credentials needed to poison a widely used downstream package.

Note: The attacker's exact access pathway for obtaining LiteLLM's PyPI publishing credentials has not been fully confirmed publicly as of this writing. The Trivy link is the working hypothesis based on available evidence.

Payload Behavior

Analysis of the malicious packages reveals a multi-stage payload. The detailed breakdown below is drawn from public research by Upwind, Wiz, Sonatype, and FutureSearch.

Execution mechanism

Version 1.82.7 contained a malicious payload injected into proxy_server.py. Version 1.82.8 added a .pth file (litellm_init.pth) to the package. Python automatically processes .pth files on every interpreter startup when the package is installed, so the payload executes silently regardless of whether litellm is explicitly imported.

A reported side effect of the .pth-based execution mechanism in 1.82.8 was a recursive fork condition (the spawned subprocess re-triggered the .pth file), which in some environments caused a process storm. This was an unintended bug in the malware, and it was one of the first observable symptoms reported by the engineer who discovered the attack publicly.

Stage 1 - Credential harvesting

The payload collected a broad range of sensitive files and data from the host, including:

• Environment variables and shell history
• SSH private keys and configuration files
• Cloud provider credentials: AWS (~/.aws/credentials, IMDS), GCP (application_default_credentials.json), Azure (~/.azure/)
• Kubernetes service account tokens and kubeconfig files; the malware also queried the Kubernetes API to enumerate secrets across all namespaces
• .env files and application secrets matching patterns like api_key, api_secret, access_token
• CI/CD and infrastructure configs: Terraform state files, Docker registry auth, GitLab CI, Jenkins, Ansible
• Database credentials for PostgreSQL, MySQL, Redis, MongoDB
• SSL/TLS private keys
• Cryptocurrency wallet files
  
  

Stage 2 - Encrypted exfiltration

Collected data was encrypted using AES-256-CBC with a random session key, which was itself encrypted with a hardcoded RSA-4096 public key. The encrypted archive was then sent via HTTPS POST to models.litellm.cloud - a domain that is not affiliated with BerriAI or the legitimate litellm project. Only the attacker holding the corresponding RSA private key can decrypt the exfiltrated data.

Stage 3 - Persistence and lateral movement

The payload installed a persistence backdoor on the local host:

• Script path: ~/.config/sysmon/sysmon.py
• Systemd user service: ~/.config/systemd/user/sysmon.service (auto-restarts every 10 seconds)
• The backdoor polls a second attacker-controlled domain (checkmarx[.]zone) approximately every 50 minutes for a URL and executes whatever binary is hosted there, providing the attacker with persistent arbitrary code execution.

If a Kubernetes service account token with sufficient RBAC permissions was present, the malware attempted to deploy privileged pods (using alpine:latest) to every node in kube-system, mounting the host filesystem to replicate the persistence backdoor at the host level.

How to Detect This Attack

Below are concrete detection signals. These are based on the known payload behavior documented by Upwind, Wiz, and Sonatype, mapped to the types of telemetry available in most enterprise environments.

Network telemetry

• Outbound HTTPS POST to models.litellm.cloud from any process - particularly Python processes. This domain is not part of legitimate litellm infrastructure.
• Outbound connections to checkmarx[.]zone (C2 polling domain for the persistence agent).
• HTTP requests to 169.254.169.254 (AWS IMDS) or metadata.google.internal originating from Python processes that have no prior history of cloud metadata access.

DNS-level blocking of models.litellm.cloud and checkmarx[.]zone will prevent exfiltration and C2 polling on affected hosts that have not yet communicated with these endpoints.

Endpoint / file system telemetry

• Creation of litellm_init.pth in any Python site-packages directory during or after a pip install operation.
• Presence of ~/.config/sysmon/sysmon.py or ~/.config/systemd/user/sysmon.service.
• Presence of /tmp/pglog or /tmp/.pg_state.
• Writing of a tar.gz archive to a temp directory from a Python subprocess shortly after package installation.
  
  

Process telemetry

• Python subprocess spawning additional Python child processes immediately following a pip install - especially recursive or high-frequency spawning patterns.
• Python processes spawning shell commands to read SSH keys, cloud credential files, or enumerate environment variables.
• Unexpected systemctl --user enable commands issued by a Python process.
  
  

Kubernetes / cloud telemetry

• API calls listing secrets across all namespaces from a service account that does not normally require this access.
• Pod creation in kube-system matching the pattern node-setup- using the alpine:latest image with hostPID: true and hostNetwork: true.
• Calls to AWS Secrets Manager ListSecrets or GetSecretValue, or SSM Parameter Store, from unexpected process lineages or IAM roles.

Detection note: Detection of this attack relies primarily on runtime behavioral signals - not static package scanning. By the time a malicious package version appears in a CVE feed or is yanked from PyPI, it has already executed in many environments. Network egress monitoring, process behavior analytics, and file integrity monitoring are the layers that catch it in time.

Indicators of Compromise (IOCs)

The following IOCs are drawn from LiteLLM's official security advisory, Upwind's payload analysis, and Wiz's campaign research.

Who Is Affected

Per LiteLLM's official advisory:

• Affected if: You installed or upgraded litellm via pip.
• Affected if: You ran pip install litellm without a pinned version and received v1.82.7 or v1.82.8.
• Affected if: You built a Docker image during this window with an unpinned pip install litellm.
• Affected if: A dependency in your environment pulled in litellm as an unpinned transitive dependency (e.g., through an AI agent framework or MCP server).
• Not affected: Customers running the official LiteLLM Proxy Docker image. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages.
• Not affected: Environments using litellm versions prior to 1.82.7 or v1.83.0 and above (the first clean release after the incident).
  
  

Remediation Steps

If you believe you may be affected, take these steps. Simply removing the package is not sufficient - the malware is designed to establish persistence and may have already communicated with attacker infrastructure.

4. Rotate all credentials on affected systems

Assume any credential present on an affected machine is compromised. This includes: SSH keys, AWS access keys and IAM credentials, GCP ADC tokens, Azure service principal credentials, Kubernetes service account tokens, API keys stored in .env files, database passwords, LLM provider API keys.

5. Block network IOCs

Block models.litellm[.]cloud and checkmarx[.]zone at your DNS resolver and firewall. Review proxy/firewall logs for any prior POST connections to models.litellm.cloud.

6. Reinstall from a verified version

Install litellm v1.83.0 or later, released via the project's new CI/CD v2 pipeline with improved security controls. LiteLLM's security advisory includes SHA-256 checksums for verified safe versions going back to v1.78.0.

7. Audit transitive dependencies

Check whether any AI agent frameworks, MCP servers, or orchestration tools in your environment pull in litellm as an unpinned transitive dependency. Treat any unpinned transitive dependency as a potential exposure vector.

Broader Context: The AI Supply Chain as Attack Surface

The LiteLLM compromise is part of a deliberate pattern. TeamPCP has consistently targeted security-adjacent and AI-adjacent tooling - a vulnerability scanner, infrastructure analysis extensions, and now an LLM proxy library. These tools share a common characteristic: they run with broad access because their legitimate function requires it.

AI libraries like litellm are particularly valuable targets because they are installed across developer workstations, CI/CD pipelines, and production environments simultaneously - often with access to credentials spanning every AI service provider an organization uses. As agentic AI systems become more prevalent, the libraries that power them will carry even more sensitive context.

The attack also illustrates the fragility of implicit trust in the open-source supply chain. LiteLLM's compromise was downstream collateral from the Trivy breach. Credential rotation that is not atomic - not simultaneous across all systems - can leave a window that sophisticated attackers are prepared to exploit.

Organizations adopting AI tooling at speed should apply the same supply chain security discipline they apply to production software: version pinning, dependency lock files, SBOM generation, private package mirrors, and runtime behavioral monitoring.

Longer-Term Mitigations

• Version pinning: Pin all AI/ML library versions in requirements.txt and lock files. Unpinned dependencies are the primary enabler of this class of attack.
• Package integrity verification: Compare SHA-256 hashes of installed packages against known-good manifests. Tools like pip-audit can help catch deviations.
• Egress controls on build environments: Python processes in build pipelines should operate under egress-controlled network policies. Unexpected outbound HTTPS connections from install-time subprocesses are a detection signal.
• Kubernetes RBAC hygiene: Development-tier service accounts should not be able to list secrets across namespaces or create pods in kube-system. Apply namespace isolation with NetworkPolicy objects.
• SBOM generation and monitoring: Generate and diff SBOMs at every build. A new .pth file added to a previously stable package manifest would be immediately visible.
• Isolate AI tooling environments: Isolate AI experimentation environments from production credential stores. Developers exploring new AI tooling should not have direct access to production API keys or cloud credentials.
  
  

Where Netenrich Comes In: Behavioral Hunting in Practice

Netenrich's detection philosophy, the foundation of Autonomous security operations, is built around a simple but important distinction: automate the known, discover the unknown. Known detections - matching a malicious package version, flagging a listed IOC domain, finding a specific file hash - are necessary but not sufficient. They only fire after the threat is already documented. What catches a campaign in motion, before the IOC list exists, is behavioral detection: noticing that something a node has never done before is now happening, at the right time, with the right context.

The LiteLLM attack is a good illustration of this in practice. Below is the sequence of behavioral signals that surfaced across monitored environments, organized by detection type.

Thesis: Known detections (IOC matches, version signatures, rule-based single-event alerts) are the floor, not the ceiling. The detections that matter in a live supply chain attack are the ones that fire on behavior that has never been seen before from that node - without requiring a prior IOC match.

Layer 1 - Automate the Known: Single-Event and Multi-Event Rule-Based Detections

These detections fire when a specific observed event matches a known-bad indicator. They are important, reliable, and should run continuously - but they depend on the threat already being documented.

Layer 2 - Discover the Unknown: Behavioral Detections Without Prior IOC Knowledge

These are the detections that would have fired even if the IOC list for this campaign had never been published - because they are based entirely on behavioral deviation from a known baseline. Each one represents something a node has never done before, in a pattern that is anomalous regardless of whether the destination domain or file hash is recognized.

Why this matters: The malicious packages were live for approximately 40 minutes before public disclosure. Known IOC-based detections can only fire after the community has documented and distributed the indicators. Behavioral detections based on what a node has never done before are the only layer capable of firing within that window.

Behavioral signal 1: Python process makes first-ever outbound POST to an external domain at install time

What happened: Immediately following a pip install litellm command, a Python subprocess initiated an HTTPS POST request to an external domain. This was the first time this process lineage had made an outbound POST to any external host from this node.

Why it fired without IOC knowledge: This node had no prior history of Python processes making outbound POST requests at install time. Package installation is expected to pull packages from PyPI - not to POST encrypted binary payloads to external endpoints. The combination of process context (Python subprocess spawned by pip), direction (outbound), method (POST), and payload type (encrypted binary, not JSON or form data) was anomalous against this node's baseline behavior, regardless of the destination.

Detection Summary: Known vs. Unknown

The table below maps each detection signal to its type and the earliest point at which it could fire in the attack timeline.

Key takeaway: Five of the eleven detections above require no prior knowledge of this campaign. They fire on behavioral deviation alone - a node doing something it has never done before, in a context that has no legitimate explanation. These are the detections that operate inside the disclosure window, before the IOC list exists. Known detections extend coverage once the campaign is documented. Both layers are necessary; neither is sufficient alone.