DevOps

Self-Host Azure DevOps Agent on AKS

Posted on by Mr. Exojo

Azure DevOps requires compute to run pipelines. It can be hosted by Microsoft (as Azure DevOps service bill) or self hosted.

Since the Docker deprecation as running execution inside Kubernetes, you can not use docker commands for your pipelines. However, if you are using Azure Container Registry from your Azure DevOps pipelines, you can execute their commands related from the Azure CLI.

This repo contains the necessary files to deploy a self-hosted Azure DevOps agent on Azure Kubernetes Service (AKS) using a Personal Access Token (PAT) authentication, next to a quick custom to enable Azure CLI for your pipelines.

Dockerfile

The Dockerfile sets up an image based on Ubuntu with several required utilities, as well as the Azure CLI. The Dockerfile creates a new user named “agent”, copies a startup script (start.sh) into the image, and sets it as the entrypoint.

FROM ubuntu:22.04

RUN apt update \

&& apt upgrade -y

RUN apt install -y curl git jq libicu70

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

ENV TARGETARCH="linux-x64"

WORKDIR /azp/

COPY ./start.sh ./

RUN chmod +x ./start.sh

RUN useradd -d /azp agent \

&& chown -R agent ./

USER agent

ENTRYPOINT ./start.sh

Kubernetes Deployment Configuration

The k8s/devopsagent-deployment.yml file is a Kubernetes deployment configuration that specifies how to deploy the Docker image on AKS. It defines the necessary resources, such as the number of replicas, the Docker image to use, and the necessary environment variables.

Startup Script

The start.sh file is a shell script that is run when the Docker container is started. It is responsible for starting the Azure DevOps Agent.

#!/bin/bash
set -e

if [ -z "${AZP_URL}" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "${AZP_TOKEN_FILE}" ]; then
  if [ -z "${AZP_TOKEN}" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE="/azp/.token"
  echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

unset AZP_TOKEN

if [ -n "${AZP_WORK}" ]; then
  mkdir -p "${AZP_WORK}"
fi

cleanup() {
  trap "" EXIT

  if [ -e ./config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan="\033[1;36m"
  nocolor="\033[0m"
  echo -e "\n${lightcyan}$1${nocolor}\n"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "${AZP_TOKEN_FILE}") \
    -H "Accept:application/json;" \
    "${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")

if [ -z "${AZP_AGENT_PACKAGE_LATEST_URL}" -o "${AZP_AGENT_PACKAGE_LATEST_URL}" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account "${AZP_URL}" is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS "${AZP_AGENT_PACKAGE_LATEST_URL}" | tar -xz & wait $!

source ./env.sh

trap "cleanup; exit 0" EXIT
trap "cleanup; exit 130" INT
trap "cleanup; exit 143" TERM

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "PAT" \
  --token $(cat "${AZP_TOKEN_FILE}") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

chmod +x ./run.sh

# To be aware of TERM and INT signals call ./run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh "$@" & wait $!

Required Azure configuration

Before the deployment, you need to enable some previous configurations:

from Azure:

  • Azure Container Registry write permissions
  • Azure Kubernetes Service admin permissions over namespace used by pipelines.

from Azure DevOps:

  • Azure DevOps Agent pool
  • Personal Access Token

On Azure Kubernetes Service:

  • Kubernetes Namespace for running ADO client
  • Write permissions for the user/group owner of PAT on the Kubernetes namespace used by pipelines.
  • Kubernetes Rolebinding for the ADO agent service account generated to the namespace used by pipelines.

ADO Agent Deployment

Example steps for a ubuntu22 tag

  1. Build and tag the image from local
docker build --no-cache --tag devopsagent:u22 --file ./Dockerfile . docker tag devopsagent:u22 myazcregistry.azurecr.io/devopsagent:u22
  1. Login with your Azure account and push the image
az login --use-device-code az acr login --name myazcregistry.azurecr.io docker push myazcregistry.azurecr.io/devopsagent:u22
  1. Create the kubernetes secret with the PAT
kubectl create secret generic devops-agent \ 
--from-literal=AZP_URL=https://dev.azure.com/yourOrg \ 
--from-literal=AZP_TOKEN=YourPAT \ 
--from-literal=AZP_POOL=NameOfYourPool -n devops
  1. Deploy the replicaset by the deployment file
kubectl apply -f k8s/devopsagent-deployment.yml -n devops

Once deployed, if the pipelines gonna deploy on a different namespace of self agent, is required allowing permissions to the default service account and for the agent service account generated:

  • Example:
kubectl get serviceaccount -n devops 
NAME SECRETS AGE 
azdev-sa-a6a8e1 1 7d6h 
default 1 17d 
kubectl create rolebinding rolebinding-sa-azdev  \
--role=apps-admin --serviceaccount=devops:azdev-sa-a6a8e1 \
--serviceaccount=devops:default -n apps

Wrapping up

This was a quick resume about the requirement of compute when you are using a Azure Kubernetes Service and you want to use their compute capacity to run Azure DevOps pipelines in your continuous integration/deployment processes.

Userful? thanks for share it