Hacking Azure DevOps

Introduction

While this case is not a particularly new one and has been posted by Matt Cooper on devblogs.microsoft.com back in August 2020. I still feel that in relation to the possible data spillage it has not received sufficient exposure and the correct amount of awareness I would have expected. I actually stumbled upon this case by accident when playing with the Azure DevOps Library variables API.

So in this post I want to showcase how a possible attacker can use a compromised developers environment to gain access to almost all the data present in an Azure DevOps Organization. While access of the developers environment in question is limited to just a single Azure DevOps Project.

Goal

So what’s the goal of this post?
The goal of this post is to emphasize the importance of zero trust and least privilege. And create awareness that security is everyone’s job.

How am I going to achieve this goal?
By demonstrating a rather unconventional & unexpected Lateral Movement Attack which utilizes a possible configuration vulnerability in an Azure DevOps Organization.

Current situation

For this showcase I have created an example Azure DevOps Organization called DemoJev. This organization is currently populated by 5 projects. As shown in the following image.

Lets assume that there are 3 developers Jan, Abdul and Henk who are working on the My Shuttle Project. They have access to:

• The project
• The code repo
• The variable group(s)
• The pipeline(s)

And indirect access to the Azure resources deployed by the pipelines. The Azure permissions and configuration are actually not relevant within this showcase and are just added to draw a complete picture of a minimalistic set-up for a cloud native application development set-up. The drawn picture is as follows.

Since it is not the question if an environment is compromised but when. It is possible to assume that the development environment of one of the developers will eventually be compromised. In this showcase the development environment of Henk has been compromised.

Henk is hacked!

It seems he is :-(

To better understand the implications of the hack lets zoom in on Henk’s permissions. Like stated in the previous chapter Jan, Abdul and Henk are all team members on the project and thus it’s save to assume that all of them have contributor role within the My Shuttle project. Assuming that out of the box permissions of the Azure DevOps project are present, Henk should at least have the following permissions;

• Modify source code (partially depending on branch policies)
• Create & delete Git repos
• Create, delete, update and run pipeline
• Create, delete and update variable groups
• Modify board information

Expected data spillage

Judging from the above noted permissions it is presumable to expect that the attacker acquired the same level of access as Henk has. So letting Azure aside (again, not important for this case) the data spillage should be limited to the MyShuttle Azure DevOps project. As illustrated in the following image.

Actual data spillage

Contrary to the expected data spillage the actual data spillage is much larger. By fiddling with som Azure DevOps API’s and a bit of PowerShell scripting I was able to produce a data spillage the size of the whole Organization. See following the following image.

So a knowledgeable attacker can basically gain read access to pretty much all the data stored within the Azure DevOps Organization, including;

• All Azure DevOps projects within the same Organization
• All repositories in all Azure DevOps projects within the same Organization
• All values of the non secret variables in all Azure DevOps projects within the same Organization
• All Artifacts regardless of the streams are private or public

The hack

To explain how this is possible I am going to use an yaml pipeline that will execute a set of PowerShell scripts which will make calls to the Azure DevOps rest API using the pipeline identity.

Authentication header

First lets start by creating an authentication header using this pipeline identity. Azure DevOps pipelines have a set of predefined variables available for use during run-time, these contain all kinds of handy information. One of these is the $(System.AccessToken) or when using PowerShell $($env:SYSTEM_ACCESSTOKEN). This variable contains the mentioned pipeline identity token. So using this token it is possible to craft an authentication header which will contain the security context of the pipeline identity token. In PowerShell this will look something like shown in the following snippet.

List all Azure DevOps projects

The first step is to list all Azure DevOps projects within the DemoJev Organization. For this I used the Projects - List API . See following snippet.

Putting the projects API call to practice

To call the API I have incorporated the authentication header into the API call by simply passing as -Headers $httHeaders. I am not going into details about the yaml pipeline as it should be pretty straight forward. However it is important to note the last two lines of this snippet. These two lines allow for the pipeline identity token to be passed to the task in question. Without this addition the token wont be accessible within the task.

Making the system token available to a Azure DevOps task.

The API call in practice

When executed, the pipeline should displaying all Azure DevOps Projects and their ID’s. As shown in the following image.

Get all Git repo’s

Now that all the project information is available, it is possible to dig a little deeper and retrieve git repositories from one or even from all of the retrieved projects. To keep the showcase simple the following snippet gets the repos of a single project, namely; MyHealthClinic. First I call the Repositories - Get API to get metadata about all the repositories present within this project. Then it is just a matter of iterating trough the response and executing a git clone command fo each repo. Most interesting part here was composing the @gitUrl variable value. As it is needed for input when calling the git clone command. To achieve this I was able to reuse the $(System.AccessToken) as a credentials object when using the ‘git clone’ command.

Putting the Git API call to practice

When everything is put together, the pipeline is as show in the following snippet.

When executed, the pipeline should result in all the present Git repositories within the project cloned to the working directory specified by the $reposDir variable. Check the following image for the result.

Get all variable values

With the project information still available it is also possible to get additional data for each of the projects. Personally next to actually cloning the repositories I found that the possibility to actually get variable data the most ‘interesting’ find. So first a call to the Variablegroups - Get Variable Groups API. This API expects a groupName (documentation states it does not, but it does!) so to get all the groups I simply tried: groupName=*. It actually worked!

Putting the variable groups API call to practice

When everything is put together, the pipeline is as show in the following snippet.

When executed, the pipeline should return the variables as a json object. It appears that the variables returned including the actual values as long as they are not marked as secrets. For variables which are marked as secrets the value is set to null. Check the following image.

What’s the vulnerability?

It turns out that all the pipelines are executed using a single Azure DevOps identity named Project Collection Build Service ({you organization name}). That’s right, this identity has read permissions to pretty much everything within all Azure DevOps Projects. The specific permissions of this identity are already covered by Microsoft docs page so I am not going to go into details about them.

To me this implementation is similar to having an application that consists out of multiple services and during installation configuring all the services to share a single service account which also has read permissions to the whole system.

The fix

Thankfully Microsoft already provided one, it is just not enabled for all Organizations. presumably Microsoft identified this as an issue somewhere in the beginning of 2020 since all the Azure DevOps Organizations created after May 2020 automatically have the fix enabled.

The fix is also quite simple when it comes to implementation. A Project Collection Administrator of the Organization in question should simply enable 3 configuration settings by following these steps:

1. Open Organization Settings
2. Under Pipelines click on Settings
3. Enable the following settings;
   1. Limit job authorization scope to current project for non-release pipelines
   2. Limit job authorization scope to current project for release pipelines
   3. Limit job authorization scope to referenced Azure DevOps repositories

NOTE: There is no save button so the settings are have an immediate effect

The moment these settings are enabled the next pipeline that is started will not use the Project Collection Build Service. Instead each project is provisioned with it’s own build service. This service has project scoped permissions and thus when the scripts as described in the The hack chapter are executed the information from the project in question is returned and nothing more. Like illustrated in the following image.

Solution implications

While implementing the fix is very straight forward it does have some implications. As of writing this post this is what I have identified as issues and/or side effects.

• If a pipeline references other repositories in addition to the repository where the pipeline resides. The repository owner must provide permissions in the pipeline for each repository in question. Even for repositories that are part of the same Azure DevOps Project.
• Permissions to organization scoped Artifacts feeds must be provided manually for each Project by navigating to the feed in question from within the project in question and clicking the Allow project-scoped build button.
• Any custom permissions which where given to the Project Collection Build Service in any project must now be also given to the individual project build service.

The Good, The Bad & The Ugly

To sum up my take on this subject here is an overview of The Good, The Bad & The Ugly.

I hope that you have enjoyed reading the read and perhaps even learned something new. If you have any questions on the subject just reach out to me via twitter on @devjevnl.