All commercial software development depends on CI/CD pipelines to build, test, and deploy code quickly. The systems within allow organizations to automate many of the administrative and logistical aspects of writing software, saving time and reducing the potential for human error.
However, the same components that make the automation in CI/CD pipelines possible also make them attractive targets for attackers.
CI/CD environments join a lot of critical infrastructure—and the credentials needed to authenticate that infrastructure—in a seamless workflow to automate the development process (hence, a pipeline).
When an attacker gains control of one or more systems within a pipeline, they may obtain access to source code, cloud infrastructure, deployment credentials, signing keys, or the production network environment it lives within. In some cases, adversaries can even turn the pipeline into a conduit for their own use, leveraging their victim’s established downstream software delivery processes to spread malicious code even further.
In this article, we’ll explain the vulnerable components of CI/CD pipelines and explore how to secure these uniquely valuable and dangerous cornerstones of software development enterprises.
What is a CI/CD pipeline?
CI/CD stands for Continuous Integration and Continuous Delivery or Continuous Deployment: industry terms describing the practice of automating the process of building, testing, and releasing software. Pipelines encompass the hygienic and management aspects of software development, aligning them on a repeatable framework that allows developers to focus more of their time on writing code.
According to the OWASP CI/CD Security Cheat Sheet, a typical pipeline begins when developers commit code to a repository and ends with software being tested and deployed to higher environments or production systems. Virtually all enterprise software is developed through a CI/CD pipeline.
A typical workflow involves developers pushing code to a source control platform. Automated tools then perform builds, execute tests, scan for vulnerabilities, create artifacts, and finally deploy the code as applications. Teams across software engineering, DevOps, cloud operations, and security frequently interact with and contribute to these systems.
To perform their functions, CI/CD environments require extensive access across an enterprise network. Pipelines often need permissions to:
- Access source code repositories
- Deploy workloads into cloud environments
- Manage containers and Kubernetes clusters
- Access secrets and environment variables
- Sign software artifacts
Common CI/CD platforms include GitHub Actions, GitLab CI/CD, Jenkins, Azure DevOps, CircleCI, and Bitbucket Pipelines, among many others. While each platform differs in implementation, they share a common characteristic: They act as trusted automation systems with privileged access across large portions of the technology stack.
Why attackers target CI/CD pipelines
Access to internal resources is intrinsic to a CI/CD pipeline, making the incumbent systems and hosting services incredibly valuable targets for attackers.
A CI/CD platform provides access to invaluable source code, supporting infrastructure, and production development systems. Additionally, services with pipelines are constantly authenticating to the systems within, and so use a cornucopia of sensitive secrets, including but not limited to:
- Access tokens and API keys
- Cloud credentials
- Service and developer user account login credentials
Many of these secrets authenticate with elevated permissions, increasing their potential and value for malicious misuse. They may be authorized for any number of arbitrary, privileged actions like creating new cloud infrastructure, publishing software packages, deploying production workloads, or authenticating into mission-critical systems.
What are attackers actually attacking?
To do its job, a CI/CD pipeline needs access to valuable hosts and services that connect to one another to form its automated code deployment ecosystem.
The tangible, real targets within are typically:
- Source code repositories, where attackers can steal a business’ code or inject their own malicious code.
- Secrets and credentials, including cloud API keys, service account tokens, SSH keys, and database credentials that can be used to access other systems. Secrets may be stolen from improperly secured secret stores or even from source code with secrets hard coded within.
- Build servers and runners, which execute code and build tasks automatically, often containing sensitive data.
- Code deployment systems, which provide direct access into production environments and cloud workloads.
- Cloud infrastructure credentials, which may allow attackers to create, modify, or destroy the resources that host the pipeline’s services.
- Code-signing infrastructure, which can be abused to make malicious software appear legitimate and trusted.
- Update channel mechanisms, enabling attackers to distribute malicious code to customers through normal release processes.
Gaining access to one of these targets often leads to the compromise of other targets in the pipeline, or the network beyond them, since they are all interdependent.
What happens after a pipeline is compromised?
Compared to any arbitrary network or resource compromise, a compromised pipeline is uniquely dangerous. Not only do attackers have access to a part of a network, but they have access to the cornerstone of that network’s commercial software production. Depending on the scope and success of the compromise, attackers may attempt any of the following:
- Steal source code and intellectual property
- Exfiltrate credentials, secrets, or sensitive data
- Deploy malware into production environments
- Modify source code maliciously
- Publish and distribute malicious software packages downstream
- Establish persistent access to the network within the pipeline’s infrastructure
- Move laterally into cloud environments
The potential for malicious impact downstream makes CI/CD environments especially valuable. If attackers compromise a pipeline that distributes software updates to customers, they may affect hundreds, thousands, or even millions of users. The owners of the compromised pipeline become unwitting accomplices of a software supply chain attack against their own customers.
One of the most notable examples of such an attack was the SolarWinds compromise.
Attackers gained access to the company's software build environment and inserted malicious code into legitimate software updates, which were then digitally signed and distributed to approximately 18,000 customers through normal update channels.
The incident demonstrated how compromising a trusted software delivery process can provide attackers with access to numerous downstream organizations without requiring separate compromises of each target.
How attackers compromise CI/CD pipelines
Attackers use multiple techniques to gain access to CI/CD systems.
Attacking a pipeline often involves the same strategies as attacking any arbitrary system: credential exposure and theft, abuse of excessively privileged permissions, compromise of a third party, or abuse of the pipeline itself.
Credential and token theft
Stolen credentials remain one of the most common attack vectors in network compromises, including CI/CD pipelines. The services running within a CI/CD pipeline rely on API tokens, SSH keys, service accounts, and cloud login credentials to authenticate with the systems within the pipeline. For example, Jenkins needs access to a code repository to be able to test new software builds, and so will have its own login credentials to the repository (or, in environments with poor security hygiene, use a developer’s credentials).
If these secrets are exposed or stolen, attackers can use them to authenticate directly to CI/CD systems.
Excessive permissions
Overly permissive access controls inflame the potential damage of credential theft. A compromised password or token does not necessarily spell doom for an entire pipeline if the permissions associated with those credentials are specifically allocated and maintained.
If the account Jenkins uses to access source code is limited to only the repositories it needs, the scope of compromise is likewise limited to those repositories. But to make life easier for themselves, a developer might forgo creating and managing several accounts with different levels of access and instead grant Jenkins broad access to all repositories. Now, if that account is compromised, so is the entire code inventory.
Third-party dependency compromise
Virtually all modern software includes third-party dependencies, whether they are packages, plugins, or software libraries. They’ve become a ubiquitous part of the development process.
One of the first actions a developer takes when starting a new project is to add all the “import” statements targeting the dependencies they’ll be relying on. A compromise of any of those dependencies puts the entire organization at risk.
OWASP classifies dependency chain abuse as a major risk category. Once incorporated into a pipeline, these dependencies may perform arbitrary malicious actions.
The Codecov breach demonstrated how third-party compromise can specifically affect CI/CD pipelines. Attackers gained access to a script Codecov owned (through credential theft) that thousands of their customers subsequently integrated into their own pipelines. Those customers found themselves victims of the now-malicious script’s actions (which led to more credential theft).
Poisoned pipeline execution
OWASP identifies Poisoned Pipeline Execution (PPE) as one of the most significant CI/CD risks. This attack occurs when adversaries gain access to a source control system.
They’ll then modify the pipeline's execution nodes, which compile, test, and package code, to execute malicious code with the already-trusted permissions assigned to the source control environment.
The mechanism of the Codecov breach is an example of such poisoning. Their maliciously edited script was typically automatically executed by customers as a shell command.
How to secure CI/CD pipelines
Effective CI/CD pipeline security requires many of the same security practices needed to secure any network environment: prudent access controls and a hygienic approach to third-party dependencies.
Applying the accompanying Zero Trust tips atop any of the practices listed below will ensure the tightest security.
Implement least privilege
Every service account, workflow, pipeline, automated job, and user should have only the permissions necessary to perform its function. Avoid granting repository-wide access when more limited permissions are sufficient. To avoid the operational overhead of maintaining many accounts with varying permissions, store them in a secrets manager.
Use an identity provider (IdP) or another conditional access tool to verify and enforce additional conditions (such as the incoming IP address) on every authentication attempt.
Maintain patching and updates
Staying up to date with the latest security patches remains one of the best assurances against compromise. Build servers, orchestration platforms, plugins, runners, and supporting infrastructure must be updated or patched regularly. Otherwise, vulnerable CI/CD components can provide attackers with direct entry points into critical systems.
Zero Trust tip: Deploy a patch management tool that continuously monitors the systems and software of the CI/CD pipeline and configure it to apply missing patches automatically during maintenance windows.
Deploy a patch management tool that continuously monitors the systems and software of the CI/CD pipeline and configure it to apply missing patches automatically during maintenance windows.
Rotate credentials and centralize secret management
Use centralized secret management solutions instead of hard-coding credentials within code repositories or configuration files.
Organizations should:
- Rotate secrets regularly with a secrets manager
- Eliminate hardcoded credentials
- Use short-lived tokens whenever possible
- Monitor when secrets are checked out of
CISA recommends minimizing usage of long-term credentials and adopting the strongest authentication methods available throughout CI/CD environments.
Zero Trust tip: Implement just-in-time (JIT) access with a system that automatically grants elevated permission only at the moment it is needed and revoke it through an automated expiration timer.
Implement just-in-time (JIT) access with a system that automatically grants elevated permission only at the moment it is needed and revoke it through an automated expiration timer.
Verify third-party packages and sources
To avoid accidentally importing or integrating a compromised third-party dependency, organizations should establish controls and mechanisms that verify their security and integrity.
This includes:
- Lock source code to use only specific, previously vetted dependency version numbers (known as “version pinning”)
- Scan source code for external dependencies and any vulnerabilities they might host with software composition analysis (SCA) tools. Configure a pipeline’s package manager to automatically review and verify the origin (known as provenance) of any third-party packages
- Maintain a software bill of materials (SBOM) of all components and dependencies used by an application.
- Review the release notes associated with upcoming updates of third-party components before allowing them to deploy.
- If a third-party dependency executes within the pipeline itself, test it in an isolated environment before integrating it into production CI workflow.
Zero Trust tip: Implement application control to restrict which software can execute within your build environment, typically by enforcing policies that allow or block specific application files.
Implement application control to restrict which software can execute within your build environment, typically by enforcing policies that allow or block specific application files.
By limiting software build environments to approved tools and binaries, organizations can reduce opportunities for malicious code execution and persistent access after the fact.
FAQs
What is CI/CD pipeline security?
CI/CD pipeline security refers to the policies, technologies, and processes used to protect the systems that build, test, and deploy software from unauthorized access, tampering, credential theft, and supply chain attacks.
How do attackers compromise CI/CD pipelines?
Attackers commonly compromise pipelines through stolen credentials (especially those with excessively broad permissions) and malicious third-party components and dependencies.
What are the most common CI/CD security risks?
Common risks include credential theft, poisoned pipeline execution, excessive permissions, and third-party component compromise.
Can a CI/CD attack become a software supply chain attack?
Yes. If attackers compromise a pipeline and use it to distribute their own malicious code or software artifacts to customers, partners, or downstream users, the incident becomes a software supply chain attack.
How can organizations secure CI/CD pipelines?
Organizations can improve CI/CD pipeline security through least-privilege access controls, credential rotation, centralized secret management, system patching, dependency verification, software configuration reviews, code signing, and application control.
How does Zero Trust improve CI/CD security?
Zero Trust reduces security risk by continuously validating identities and the resources they access or execute, enforcing least privilege, segmenting access to limit lateral movement, and containing attacks before they ever reach production systems or downstream software consumers.
