Why You Should Test Your Kubernetes Network Policies

The above example shows a Network Policy which by default selects pods in the namespace, labelled run=nginx. It also contains a single ingress rule allowing traffic from all pods in the same namespace, labelled access=true. When applying this policy to a Kubernetes cluster, it results in all traffic not specified in the ingress rule to run=nginx pods in the default namespace being blocked. There are more options when declaring Network Policies, such as specifying namespaces, CIDRs, ports or protocols, but the basic behaviour is always the same.

Except that it isn’t. This specification is how Network Policies are described by the Kubernetes docs, but the documentation also mentions that the default networking plugin does not support Network Policies. Instead, various network providers are suggested who officially support Network Policies.

Testing Suggested Network Providers

So off we go, testing our Network Policy with some of the suggested plugins. Let’s start by following the tutorial on declaring Network Policies on a fresh Kubernetes cluster with Calico installed:

So far everything works as expected. We started and exposed an nginx deployment with 2 replicas and a busybox for testing. Using wget we established that traffic can reach the nginx pods through their service. Now we apply the Network Policy from the tutorial and test the connection again:

That worked out well! The download timed out, as our policy only allows connections from pods labelled access=“true“. Let’s start another busybox with that label and ensure that its traffic can reach the nginx pods through their service:

This time the busybox’s download did not timeout, so our Network Policy is in effect as expected. Let’s try the same with a new cluster using Weave Net:

Huh, looks like Weave does not want to play as nice. We expected the last wget above to not go through, as the busybox pod was missing the correct labels. Maybe we missed something when installing it? Looking at the documentation on installing the Weave Net addon for Network Policy, it states:

„The Weave Net addon for Kubernetes comes with a Network Policy Controller that automatically monitors Kubernetes for any NetworkPolicy annotations on all namespaces and configures iptables rules to allow or block traffic as directed by the policies.“

This means that no extra installation should be required, but let’s verify it just to be sure:

The list of images for pods labelled name=weave contains weave-npc, the image for the Network Policy Controller, so it is definitely included.

Another thing standing out in above quotation is the mention of „annotations on namespaces“. This is not mentioned on the current documentation for Kubernetes Network Policies. Going through another tutorial, this time on Weave’s website, it seems like this namespace annotation feature does not work (anymore). In that tutorial the policies at least took effect, so maybe it has to do with the policies themselves.

Time for a Systematic Approach

To figure out, whether the policies are at fault, we want to test different kinds of policies. For this task we wrote a tool which analyses the Network Policies in the cluster and runs tests against them. These tests are executed by a DaemonSet whose pods hook themselves into the network namespaces of pods affected by Network Policies and then use nmap to test the connection to the policies‘ selected pods. This approach ensures that our test traffic is identical to production traffic in the cluster, without interfering with it.

To evaluate the functionality of these three plugins we used the example Network Policies from the excellent Kubernetes Network Policy Recipes. This Git Repository contains examples of Network Policies including descriptions of their effects. We extracted the manifests, setup and teardown processes from each recipe and used them as test set. In this evaluation, we skipped the recipes for external traffic as it is not required by our test tool. The table below shows all tests generated for each recipe and the results for each framework over seven test runs. A checkmark (✓) indicates that the case always was successful, a cross (❌) that it always failed and tilde (~) that it sometimes failed.

Calico’s Performance