Friction Free Kubernetes on M1 Macs
Upgrading to the M1 Pro MacBook has been a satisfying experience for the most part. Not without some stumbles, however.
If you’re a developer working with containers and Kubernetes and your images are built for amd64; it’s a rude awakening to realize that they will no longer simply work.
We are left with 2 options:
- Modify the build scripts to compile for a multi-architecture setup (amd64 and aarch64).
- Run the containers on an amd64 VM.
Modifying build scripts for a multi-architecture setup is easy if it’s your own code; but maybe hard to impossible if you’re working with third-party images.
The best of both worlds has been running 2 Kubernetes instances simultaneously — one for aarch64 containers and one for amd64 — setup identically and able to talk to each other.
Enter Lima and Colima
Lima creates Linux virtual machines on macOS with support for Intel on ARM in addition to ARM on ARM. If you’re running Ventura or higher, you can run your ARM VMs with native Rosetta support. To support Intel on ARM, Lima leverages qemu.
Colima launches Lima behind the scenes and sets up VMs with a container runtime and kubernetes. With one command you can have a kubernetes instance up and running, and that’s what we’re going to see.
Installation
It’s worth reviewing the documentation from Colima and Lima but for completeness here are the basic installation steps:
$ brew install colima
$ brew install docker
$ brew install kubectlCreating VMs with Docker & Kubernetes
Containerd is an alternative to docker; and perfectly acceptable but the steps below use Docker as the runtime engine and also as a build agent.
Create an aarch64 Kubernetes + Docker instance:
$ colima start -p k8s-aarch64 --editThis initiates the creation of an instance called k8s-aarch64 dropping us in an edit mode for it’s configuration.
Update the following configurations :
- kubernetes/enabled: true
- network/address: true
- vmType: vz
- mountType: virtiofs
Once complete check the following:
$ colima ls
PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS
k8s-aarch64 Running aarch64 2 2GiB 60GiB docker+k3s 192.168.107.4
$ limactl ls
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
colima-k8s-aarch64 Running 127.0.0.1:56822 vz aarch64 2 2GiB 60GiB ~/.lima/colima-k8s-aarch64
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* colima-k8s-aarch64 colima-k8s-aarch64 colima-k8s-aarch64
$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT
colima-k8s-aarch64 colima [profile=k8s-aarch64] unix:///Users/yourusername/.colima/k8s-aarch64/docker.sockNotes
- Note the VMType of vz — this is Rosetta support but only available from Ventura onwards. Lower OS versions will have to stay with qemu.
- network/address can be left off if you’re facing issues with getting an IP addresses. The VM will still start and you’ll be able to access it; but will be limited to port forwarding into the VM to reach its services.
- You can access the kubernetes cluster and docker cluster directly by ensuring you’re using the right context.
Create an amd64 Kubernetes + Docker instance:
$ colima start -p k8s-amd64 --editConfigurations:
- arch: x86_64
- kubernetes/enabled: true
- network/address: true
We’re looking for:
$ colima ls
PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS
k8s-aarch64 Running aarch64 2 2GiB 60GiB docker+k3s 192.168.107.4
k8s-amd64 Running x86_64 2 2GiB 60GiB docker+k3s 192.168.106.5
$ limactl ls
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
colima-k8s-aarch64 Running 127.0.0.1:56822 vz aarch64 2 2GiB 60GiB ~/.lima/colima-k8s-aarch64
colima-k8s-amd64 Running 127.0.0.1:55536 qemu x86_64 2 2GiB 60GiB ~/.lima/colima-k8s-amd64
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
colima-k8s-aarch64 colima-k8s-aarch64 colima-k8s-aarch64
* colima-k8s-amd64 colima-k8s-amd64 colima-k8s-amd64
$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT
colima-k8s-aarch64 colima [profile=k8s-aarch64] unix:///Users/yourusername/.colima/k8s-aarch64/docker.sock
colima-k8s-amd64 colima [profile=k8s-amd64] unix:///Users/yourusername/.colima/k8s-amd64/docker.sockSo, here we have 2 simultaneous instances of Kubernetes and Docker. Running a container or building a docker image on Intel or ARM is just a matter of switching contexts.
Install additional components on the Kubernetes cluster (optional)
Ensure you’re installing on the right cluster (kubectl config use-context …)
kube-dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yamlcert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yamlNGNIX Ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yamlFAQs
Where are the lima VM configuration files?
Lima VM configs are stored in the ~/.lima directory with subfolders for each VM.
Where are the colima configuration files?
Colima configs are stored in the ~/.colima directory with subfolders for each instance.
What’s the best way to change a config (say increase the memory)?
Execute colima start -p <profile name> — edit to change the config is the easiest way.
How do I log into the VM?
2 options (to log into the k8s-aarch64 VM for example):
- colima ssh -p k8s-aarch64
- limactl shell colima-k8s-aarch64
