↓Skip to main content

Scaling Jenkins Build Agents with Kubernetes Pods

·5 mins

Based on the last tutorial on how to run Jenkins inside a Kubernetes cluster it is now time to leverage the Kubernetes infrastructure to scale build-jobs across the cluster.

Sample Build Jobs #

First, two projects are needed for testing the setup. The jobs won’t do anything useful, they will just wait 10 seconds and then continue.

The jobs are being created by clicking on “New Item” in the dashboard and then by selecting “Freestyle Project”.

IMAGE 07-jenkins

In the “Build”-section the “Execute shell”-build step needs to be added.

IMAGE 08-jenkins

The command the build job shall perform is

sleep 10

IMAGE 09-jenkins

After saving, the new job will be displayed in the dashboard. Another job needs to be created until there are two:

When both jobs are triggered at the same time, they will be shown under “Build Executor Status”.

IMAGE 10-jenkins

For now, both jobs are being executed within the pod. This won’t scale in the long run.

The actual goal is to run every instance of a running build job in a newly created pod. That pods-lifecycle is tied to lifecycle of the build-job.

This means, that when a new job is triggered a new pod will be created, the job run within this pod and when the job has finished, the pod will be deleted.

To achieve this, some additional configuration has to be made.

Install Kubernetes Plugin #

The Kubernetes-plugin is installed via

  • “Manage Jenkins”
  • “Manage Plugins”

IMAGE 11-jenkins

Then

  • “Available”
  • selection of “Kubernetes”
  • pressing “Install without restart”

IMAGE 12-jenkins

Then everything will be installed:

IMAGE 13-jenkins

Adding Kubernetes Secrets #

Two Kubernetes-secrets are needed.

The first one is jenkins-robot.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-robot
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-robot
  namespace: jenkins
  labels:
    "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims", "events"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods", "pods/exec", "persistentvolumeclaims", "events"]
  verbs: ["create", "apply", "delete", "deletecollection", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-robot-binding
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-robot
subjects:
- kind: ServiceAccount
  name: jenkins-robot
  namespace: jenkins

The second secret is jenkins-robot-global.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-robot-global
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-robot-global
  namespace: jenkins
  labels:
    "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods", "pods/exec", "persistentvolumeclaims"]
  verbs: ["create", "apply", "delete", "deletecollection", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-robot-global-binding
  namespace: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-robot-global
subjects:
- kind: ServiceAccount
  name: jenkins-robot-global
  namespace: jenkins

Both secrets are applied with:

kubectl apply -n jenkins -f jenkins-robot.yaml
kubectl apply -n jenkins -f jenkins-robot-global.yaml

Those secrets needs to be added to Jenkins.

The first secret is retrieved via:

kubectl -n jenkins get serviceaccount jenkins-robot --template='{{range .secrets}}{{.name}}{{"\n"}}{{end}}' | xargs -n 1 kubectl -n jenkins get secret --template='{{ if .data.token }}{{ .data.token }}{{end}}' | head -n 1 | base64 -d -

This needs to be copied into the clipboard for it to be pasted in the next step.

In Jenkins, the credential is created via

  • Manage Jenkins
  • Manage Credentials

IMAGE 14-jenkins

Then, by pressing

  • New Item

IMAGE 15-jenkins

Two more links:

  • Global credentials (unrestricted)

IMAGE 16-jenkins

And then

  • Add credentials

IMAGE 17-jenkins

The following values have to be added:

  • Kind: Secret text
  • Scope: Global
  • Secret:
  • ID: Jenkins-Robot

IMAGE 18-jenkins

The same has to be done for the secret Jenkins-Robot-Global, which can be retrieved via:

kubectl -n jenkins get serviceaccount jenkins-robot-global --template='{{range .secrets}}{{.name}}{{"\n"}}{{end}}' | xargs -n 1 kubectl -n jenkins get secret --template='{{ if .data.token }}{{ .data.token }}{{end}}' | head -n 1 | base64 -d -

IMAGE 19-jenkins

Finally, both credentials show up in Jenkins:

IMAGE 20-jenkins

Adding the Kubernetes Cloud to Jenkins #

Jenkins needs to know that there is a Kubernetes cluster that it can use. The following is explaining how to configure that.

Add the Kubernetes cluster via:

  • Manage Jenkins
  • Manage Nodes and Clouds
  • Configure Clouds
  • Add a new cloud: Kubernetes

IMAGE 21-jenkins

  • Kubernetes Cloud Details

The following values need to be added:

  • Name: Kubernetes
  • Kubernetes URL https://:6443
  • Kubernetes Namespace: jenkins
  • Credentials: Jenkins-Robot
  • Jenkins Tunnel: :50000

The IP-address of the cluster node with role master can be retrieved with:

kubectl get nodes -o wide | grep master

The IP of the Jenkins tunnel is the IP of the service called jenkins-jnlp that can be retrieved with:

kubectl get service jenkins-jnlp -n jenkins

IMAGE 23-jenkins

The rest of the values are left with their default values:

IMAGE 24-jenkins

The connection can be tested by pressing the “Test Connection”-button.

Adding Pod- and Container-Templates #

The Pod-template needs to be filled out as follows:

Pod Template:

  • Name: jnlp
  • Usage: Use this node as much as possible

IMAGE 25-jenkins

For this tutorial, the containers for these agents are based on the jenkins/jnlp-agent-alpine

  • Name: jnlp
  • Docker image: jenkins/jnlp-agent-alpine

IMAGE 26-jenkins

A “Host Path”-volume needs to be added:

IMAGE 27-jenkins

  • Host path: /var/run/docker.sock
  • Mount path: /var/run/docker.sock

IMAGE 28-jenkins

An “Empty Dir”-volume is also required

IMAGE 29-jenkins

  • Mount path: /home/jenkins/agent

IMAGE 30-jenkins

The configuration is finalised by pressing the “Save”-button.

Finally, Jenkins needs to be configured that only agent nodes shall run build jobs. This is done via:

  • Manage Jenkins
  • Manage Nodes and Clouds
  • master
  • Configure

The following values need to be provided:

  • Number of executors: 0
  • Usage: Only build jobs with label expressions matching this node

IMAGE 31-jenkins

Again, when finished, the “Save”-button needs to be pressed.

Testing the Pod Build Agents #

When both build jobs are being triggered at the same time, it can be observed, that two pods are being launched under “Build Executor Status”

IMAGE 32-jenkins

Then, when the pods are running the job are being inside them:

IMAGE 33-jenkins

Resources #