Self-Host Azure DevOps Agent on AKS

About
Latest Posts

Miguel Ramirez Exojo is an independent Cloud, DevOps and Linux IT professional, offering related remote services with more than 17 years on IT departments and consultancy.

Dockerfile

The Dockerfile sets up an image based on Ubuntu with several required utilities, as well as the Azure CLI. The Dockerfile creates a new user named “agent”, copies a startup script (start.sh) into the image, and sets it as the entrypoint.

FROM ubuntu:22.04

RUN apt update \

&& apt upgrade -y

RUN apt install -y curl git jq libicu70

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

ENV TARGETARCH="linux-x64"

WORKDIR /azp/

COPY ./start.sh ./

RUN chmod +x ./start.sh

RUN useradd -d /azp agent \

&& chown -R agent ./

USER agent

ENTRYPOINT ./start.sh

Kubernetes Deployment Configuration

The k8s/devopsagent-deployment.yml file is a Kubernetes deployment configuration that specifies how to deploy the Docker image on AKS. It defines the necessary resources, such as the number of replicas, the Docker image to use, and the necessary environment variables.

Startup Script

The start.sh file is a shell script that is run when the Docker container is started. It is responsible for starting the Azure DevOps Agent.

#!/bin/bash
set -e

if [ -z "${AZP_URL}" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "${AZP_TOKEN_FILE}" ]; then
  if [ -z "${AZP_TOKEN}" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE="/azp/.token"
  echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

unset AZP_TOKEN

if [ -n "${AZP_WORK}" ]; then
  mkdir -p "${AZP_WORK}"
fi

cleanup() {
  trap "" EXIT

  if [ -e ./config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan="\033[1;36m"
  nocolor="\033[0m"
  echo -e "\n${lightcyan}$1${nocolor}\n"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "${AZP_TOKEN_FILE}") \
    -H "Accept:application/json;" \
    "${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")

if [ -z "${AZP_AGENT_PACKAGE_LATEST_URL}" -o "${AZP_AGENT_PACKAGE_LATEST_URL}" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account "${AZP_URL}" is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS "${AZP_AGENT_PACKAGE_LATEST_URL}" | tar -xz & wait $!

source ./env.sh

trap "cleanup; exit 0" EXIT
trap "cleanup; exit 130" INT
trap "cleanup; exit 143" TERM

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "PAT" \
  --token $(cat "${AZP_TOKEN_FILE}") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

chmod +x ./run.sh

# To be aware of TERM and INT signals call ./run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh "$@" & wait $!

Required Azure configuration

Before the deployment, you need to enable some previous configurations:

from Azure:

Azure Container Registry write permissions
Azure Kubernetes Service admin permissions over namespace used by pipelines.

from Azure DevOps:

Azure DevOps Agent pool
Personal Access Token

On Azure Kubernetes Service:

Kubernetes Namespace for running ADO client
Write permissions for the user/group owner of PAT on the Kubernetes namespace used by pipelines.
Kubernetes Rolebinding for the ADO agent service account generated to the namespace used by pipelines.

ADO Agent Deployment

Example steps for a ubuntu22 tag

Build and tag the image from local

docker build --no-cache --tag devopsagent:u22 --file ./Dockerfile . docker tag devopsagent:u22 myazcregistry.azurecr.io/devopsagent:u22

az login --use-device-code az acr login --name myazcregistry.azurecr.io docker push myazcregistry.azurecr.io/devopsagent:u22

Create the kubernetes secret with the PAT

kubectl create secret generic devops-agent \ 
--from-literal=AZP_URL=https://dev.azure.com/yourOrg \ 
--from-literal=AZP_TOKEN=YourPAT \ 
--from-literal=AZP_POOL=NameOfYourPool -n devops

Deploy the replicaset by the deployment file

kubectl apply -f k8s/devopsagent-deployment.yml -n devops

Once deployed, if the pipelines gonna deploy on a different namespace of self agent, is required allowing permissions to the default service account and for the agent service account generated:

Example:

kubectl get serviceaccount -n devops 
NAME SECRETS AGE 
azdev-sa-a6a8e1 1 7d6h 
default 1 17d

kubectl create rolebinding rolebinding-sa-azdev  \
--role=apps-admin --serviceaccount=devops:azdev-sa-a6a8e1 \
--serviceaccount=devops:default -n apps

Wrapping up

This was a quick resume about the requirement of compute when you are using a Azure Kubernetes Service and you want to use their compute capacity to run Azure DevOps pipelines in your continuous integration/deployment processes. Let me know if you need help with it o related items.

Userful? thanks for share it