↓Skip to main content

How To Setup A Private Docker Registry On K3s

·3 mins

It is not always appropriate to push ones own container images to a public registry. This post shows a quick way to create a private image registry inside a K3s Kubernetes cluster.

Please note, that with the following manifest, when the registry resources are being removed from the cluster, all images will be removed as well. There is a TODO in the very last line that addresses this.

Also important to note: The registry is unsecured. Further steps have to be taken to secure it with methods like username/password or certificates which is not scope of this tutorial.

Kubernetes Resources #

Before this manifest gets applied, the domain name under spec:rules:hostneeds to be changed accordingly.

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: docker-registry-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: registry.domain.de
    http:
      paths:
      - path: /
        backend:
          serviceName: docker-registry-service
          servicePort: 5000

---
apiVersion: v1
kind: Service
metadata:
  name: docker-registry-service
  labels:
    run: docker-registry
spec:
  selector:
    app: docker-registry
  ports:
    - protocol: TCP
      port: 5000

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  labels:
    app: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: docker-registry
        image: registry
        ports:
        - containerPort: 5000
          protocol: TCP
        volumeMounts:
        - name: storage
          mountPath: /var/lib/registry
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
      volumes:
      - name: storage
        emptyDir: {} # TODO -make this more permanent later

The manifest gets applied with:

kubectl apply -f registry.yaml

Settings #

On the K3s nodes, the file /etc/rancher/k3s/registries.yaml needs to be created with the following content. The domain name needs to match the one from above manifest.

mirrors:
  "registry.domain.de":
    endpoint:
      - "http://registry.domain.de"

I did this on server and agent nodes. I am not sure if it really has to be done on the agents, I just did.

After that, server and agents need to be restarted.

On the server execute:

systemctl restart k3s

And on the agent node:

systemctl restart k3s-agent

If the changes applied can be checked with:

crictl info

There is a section called registry that should list the newly created private registry.

The local workstation also needs to know about the new registry. I am working on macOS with Docker Desktop. Under “Preferences -> Docker Engine”, the settings had to be extended with the following entry:

{
  ...

  "insecure-registries": [
    "registry.domain.de"
  ]
}

Test the Registry #

In order to test if an image can be pushed to the new registry, I have built a small image just containing Nginx with a custom HTML-file:

<html>
<head><title>Hello World!</title>
  <style>
    html {
      font-size: 500.0%;
    }
    div {
      text-align: center;
    }
  </style>
</head>
<body>
  <div>Hello World!</div>
</body>
</html>

The Dockerfile for the new image:

FROM nginx:alpine
COPY index.html /usr/share/nginx/html

Build and tag the new image according to your registry domain:

docker build -t registry.domain.de/hello:latest .

Finally the image can be pushed.

docker push registry.domain.de/hello:latest

Hopefully that worked without any issues.

For an additional test, the image can also be removed from the local workstation and then be pulled again from the private registry.

docker rmi registry.domain.de/hello:latest
docker pull registry.domain.de/hello:latest

Resources #