How to Set Up a K3s Cluster on WireGuard

This blog post is a tutorial on how to set up a WireGuard network using Raspberry Pis for a K3s cluster, with each device connected to a central cloud virtual machine (VM). 

At the core of inovex are the values innovation and excellence. If we want to be and remain innovative and excellent in a dynamic environment, we have to take care of it. Over the years we have developed a series of measures with which we ensure our high knowledge and quality standards. One of these are our Tech Days, where colleagues from a technology department meet and train together with and on cool technologies.

Our last Tech Day was held remotely due to the coronavirus. As a socialising event, we had a hackathon with every participant getting a Raspberry Pi sent home beforehand. My group had the idea to build a K3s cluster spanning across the desks of all team members. Since we were at home and most of our home networks are behind at least one NAT device, we had to build a VPN connecting our Raspberry Pis. In this blog post I want to show our initial prototype setup with a step by step tutorial to reproduce the setup.

Initial Setup

In our prototype we decided to build the WireGuard network as a star topology: every Raspberry Pi is connected to a central cloud VM. We did this since the VM has a stable public IP and thus every Raspberry Pi is able to initiate a WireGuard tunnel to the VM. The VM will also host our K3s master.

The Raspberry Pis are hosting the K3s agents. (After checking the network ranges of the participants’ home setups to avoid ip range collisions) we chose the network 10.222.0.0/24 for our nodes in the WireGuard network.

The next to section will talk you through the setup of the cloud VM and the Raspberry Pis.

Cloud VM

This section talks you through the installation and configuration of the cloud VM. All shell commands must be executed as the root user. As a convention we start the start of a shell command with a $ in our listings. You do not have to type that into your shell.

Wireguard

We choose Ubuntu 20.04 as the operating system for our cloud VM. The first step was to install the WireGuard package:

Since we want to route traffic between the Raspberry Pis, we had to enable IP forwarding on the cloud VM

The next step is to generate the public and the private keys for our WireGuard tunnel:

With these two keys we are able to set up the WireGuard wg0 interface. Place the following config in /etc/wireguard/wg0.conf:

With this configuration in place we can bring up the wg0 interface:

After this step, your cloud VM has a network interface wg0 with the IP address 10.222.0.1 assigned. We set a static route for the network 10.222.0.0/24 to that interface:

This allows us to dynamically add new WireGuard clients to our network.

K3s

The next step is to install and configure the K3s master. We did this via the following oneliner:

We recommend replacing this with a proper installation routine in a production setup since executing foreign scripts from the internet with root privileges is a security nightmare.
After the installation has finished we have to reconfigure the flannel CNI in K3s to use the wg0 interface. Place this systemd drop in /etc/systemd/system/k3s.service.d/network.conf

and restart the k3s systemd service:

To check your K3s installation, run:

Raspberry Pis

The steps described in this section must be repeated for every Raspberry Pi you want to connect to your cluster.

In our prototype, we use the Ubuntu 20.04 image from the rasperry pi imager utility on a Raspberry Pi 4 with 2 GB RAM. Since we want to install K3s on the nodes, we had to modify the kernel command line to enable cgroup memory management. After installing the Ubuntu 20.04 image to a SD card, mount the SD card on your PC and append

to the boot/cmdline.txt file. After the first boot you should change the hostname of the Raspberry Pi to something unique, since K3s uses the hostname as kubernetes node name by default.

“(On raspbian) this can be done with the „sudo raspi-config“ command or just editing the „/etc/hostname“ file and rebooting“

Install the WireGuard package via:

Generate a public and a private key for the Raspberry Pi:

Connecting the Raspberry Pi to the cluster

To connect the Raspberry Pi to the WireGuard network and the K3s cluster we have to exchange some information.

From the Raspberry Pi we need:

• The IP address of the Raspberry Pi in the WireGuard network
• The WireGuard public key generated on the Raspberry Pi

From the cloud VM we need:

• The WireGuard public key generated on the cloud VM
• The K3s token located in /var/lib/rancher/k3s/server/token
• The public IP of the VM

Let’s assume the following information:

• IP address of the Raspberry Pi in the network: 10.222.0.106
• WireGuard public key of the Raspberry Pi: csQQ8c7waCFksyIQyCOIu/eqxaGUxueu8h02qr1f81Q=
• WireGuard public key of the cloud VM: rcflbneYW/3wVQy8H/jDi/oGlLgyrC4vmJvt4YJOVmw=
• K3s token: K10a5fe90072692d561a72e6f793ec8392748c4e9b604e81d644c755f9dd6207::server:b730afa6f2572b914be9f226fc774
• Public IP of the cloud VM: 192.0.2.20

The first step is to add the Raspberry Pi as a peer to our WireGuard network on the cloud VM:

After we have allowed the Raspberry Pi to join our WireGuard network we configure the wg0 interface on the Raspberry Pi. Place the following configuration in /etc/wireguard/wg0.conf on the Raspberry Pi:

After creating this configuration run:

Now you should be able to ping the cloud VM via the WireGuard network:

The next step is to install the K3s agent. We used this oneliner on our prototype:

Replace the K3s_TOKEN with the token generated on your cloud VM. The token is in the file /var/lib/rancher/k3s/server/token on the cloud VM.

After the installation we have to configure the flannel overlay. Create the following systemd drop in:

Then run:

Final Step

The last step is to check if the Raspberry Pi joined your K3s cluster. Check via kubectl on the cloud VM:

With this setup we managed to build a K3s cluster in half a day with 22 nodes in many cities across Germany including Hamburg, Cologne and Karlsruhe.

Pitfalls

If you use raspbian, you have to build the kernel module for WireGuard by yourself. Make sure that the correct kernel headers for the correct running kernel are used when building, especially if you have previously made an “rpi-update“.