Did you know that you can run your Azure DevOps pipeline jobs in a container? It’s a great way to combine the benefits of Microsoft Hosted Agents and Self-Hosted Agents. In this post, I will explain what Azure DevOps Container Jobs are and how you can use them to your advantage. But first, a recap of Microsoft Hosted Agents and Self-Hosted Agents.
What are Microsoft Hosted Agents
Microsoft Hosted Agents are a type of agent in Azure DevOps that are hosted and managed by Microsoft. These agents are pre-configured with certain software depending upon the type of agent (Windows, macOS, or Linux) which can run jobs targeting different platforms. With Microsoft Hosted Agents you trade off control for convenience as they are an excellent choice if you need to quickly scale your operations without worrying about the underlying infrastructure management.
Pros:
Maintenance-Free: Microsoft handles all the maintenance, updates, and upgrades, freeing up your time for other tasks.
Scalability: Microsoft Hosted Agents offer excellent scalability (check pricing
for cost details), agents are immediately available in your Organization. Simply add or remove them to your liking.
Pre-Installed Tools: These agents come with a large variety of pre-installed tools, saving you from manual installation and configuration.
Cons:
Limited Customization: These agents offer limited scope for customization, which might not be ideal for complex, tailored workflows.
Release cadence: Microsoft decides when and what tools to update on the agents, which might introduce breaking changes.
Usage Limits: There are usage limitations, which might require you to manage your build and deployment tasks carefully.
Less Control: Since Microsoft manages everything, you have less control over the environment and its settings.
What are Self-Hosted Agents
Self-Hosted Agents are agents in Azure DevOps that you manage and maintain on your own infrastructure. These agents can be installed on local machines, on-premises servers, or even on virtual machines in the cloud. With Self-Hosted Agents you trade off convenience for control as they are a great choice if you need more control over the environment and its settings, or if you need to use specific versions of tools and software.
Pros:
Greater Control: With Self-Hosted Agents, you have complete control over the environment, settings, and installed tools, providing flexibility for complex workflows.
No Usage Limits: There are no strict usage limitations, allowing for more extensive and continuous usage.
Customizable Setup: You can tailor the setup to match your specific requirements, including installing specific versions of tools and software.
Cons:
Requires Maintenance: Unlike Microsoft Hosted Agents, you are responsible for maintaining, upgrading, and updating the Self-Hosted Agents.
Infrastructure Costs: You’ll need to account for the costs of the infrastructure needed to host these agents.
Delayed Access to New Features: New features and improvements will not be immediately available, as they depend on when you update your agents.
Azure DevOps Container Jobs: The Best of Both Worlds
Azure DevOps Container Jobs operate by running each job in an isolated, consistent container which you specify in your pipeline YAML code. You can reference a container from a public or a private repository (just make sure your service connection has sufficient privileges to the repository at hand).
This means you can have the convenience and scalability of Microsoft Hosted Agents, while also having the control and flexibility of Self-Hosted Agents. Let’s take a look at the main benefits.
Customization: Like Self-Hosted Agents, container jobs can be highly customized. You can choose the base images, install specific tools and dependencies, and set environment variables. This makes it ideal for complex workflows that require specific configurations, providing the flexibility similar to Self-Hosted Agents.
Maintenance-free: Container jobs, like Microsoft Hosted Agents, are maintenance-free. Azure DevOps manages the underlying infrastructure, so you don’t have to worry about system updates and upgrades. This frees up time and resources, allowing you to focus on critical development tasks.
Consistent Environment: With container jobs, each run is executed in a clean, consistent environment. This reduces the chances of errors due to discrepancies in the environment setup, enhancing the reliability of your DevOps pipeline.
Scalability & Efficiency: Container jobs can be quickly spun up and torn down, offering excellent scalability. They are also efficient as they utilize resources effectively, reducing overheads.
Early detection: Similar to Self-Hosted Agents, container jobs allow for high level of flexibility by running the same code against multiple versions of tools. For example, you can run your code against different versions of PowerShell to detect early if your code is compatible with the latest version.
Full control: The creator of the container is in control of what versions of tools and dependencies, and set environment variables are used. This is similar to Self-Hosted Agents, providing the flexibility to tailor the agent specific requirements.
Limitations
While container jobs offer the best of both worlds, there are a few limitations to keep in mind:
Only windows-2019 and ubuntu-* images support running containers. The macOS image doesn’t support running containers.
The Red Hat Enterprise Linux 6 build of the agent won’t run container job.
Windows requires that the kernel version of the host and container match.
How to Use Container Jobs in Azure DevOps
For those who are not very familiar with containers, Microsoft hosts a container registry where you can find a variety of container images. It’s an amazing source to use as a base image and perfect for this example. The registry is called Microsoft Container Registry (MCR) and you can find it here
. I suggest using the search in the top right corner as the left hand filters are a bit limited category wise.
Now lets get started on creating a container job using an Azure DevOps pipeline
Note
The next steps assumes that the following prerequisites are met:
Azure DevOps Organization with a Microsoft hosted agent available and accessible
A project in the concerning Azure DevOps organization exists and is accessible
A ready to use Git repo in the concerning project exists and is accessible
Create a new .yaml file in the concerning repo
Add the minimal pipeline scaffolding, by copy pasting the following snippet
Now, lets extend the scaffolding pipeline with a PowerShell step that contains a simple inline script to give this pipeline a body. Copy and paste the whole following snippet or extend your existing snippet with the bottom 6 lines.
Now with the required scaffolding in place its time to create a pipeline instance and run it. To do this, navigate to the concerning repo in the Azure DevOps portal and click on the Repos tab. Then click on the Files tab and select the .yml file we created in step 1. Click on the Run button and select Run pipeline. This will create a new pipeline instance and run it. The result should be similar as show in the following image.
Run pipeline results
Notice the Linux version and the available PowerShell modules. We are going to need this as a comparison in the later steps.
Next lets add a container job to the pipeline. To do this, we need to add a container definition as noted in line 10 of the following snippet. And since we are referencing a public container registry a simple image reference as show in line 11 of the same snippet is sufficient. Make sure to extend your pipeline accordingly.
Check-in the changes and run the pipeline again. The result should be similar as show in the following image.
Run pipeline results
Notice the difference between this run and the first one. First you should see two additional steps Initialize containers and Finalize containers. Second the Linux version and the available PowerShell modules are different. This is because the all the steps part of this job are running inside the specified container, the Azure PowerShell Debian container to be precise.
Congratulations, you have just successfully created and run a container job in Azure DevOps!
How to build and use your own container with Azure DevOps container jobs
Now that you have seen and experienced container jobs in Azure DevOps, you are probable wondering how to build and use your own containers. This is a great question and I will cover this in the next post I plan to release before the end of this month.
Wrapping up
And there you have it, Azure DevOps Container Jobs, the best of both Worlds. There are plenty more configuration options available so make sure to check out the reference material used to make this post.
