Spinnaker vs. Argo CD vs. Tekton vs. Jenkins X: Cloud-Native CI/CD

With tools like Kubernetes becoming essential for most modern cloud deployments and companies adopting their take on DevOps culture, there is high demand for cloud-native CI/CD. But how can we achieve increased velocity in deployment while also taking care of things like infrastructure as code, immutability, portability and security? Many tools try to offer solutions―some have just surfaced recently, others have been around for years. In this blog post, we’ll have a look at Spinnaker, Argo CD, Tekton Pipelines and Jenkins X.

 

Introduction

All right, we’re gonna try and compare tools. Why try? When we talk about continuous integration (CI) and continuous deployment (CD), a myriad of tools is available, and each of them has a slightly different scope. Jenkins for example has been around for years and offers great pluggability, leading to a highly customizable CI/CD experience―you can build almost anything (both in a good and a bad way). Argo CD in contrast is roughly one year old at the time of writing. It serves the very specific use case of syncing a Git repository with a Kubernetes cluster. This blog post is neither complete nor decisive on a winner because there is no “one size fits all“ approach to cloud-native CI/CD. We merely want to highlight tools you might have never heard of and revisit popular tools you probably use already. If this article can serve as an entry point for your research on which CI/CD tool to try in your next project, we reached our goal.

The Contestants

At inovex we are always on the lookout for tools that solve problems. Therefore, we selected the following tools to be featured in this post:

• Spinnaker, compared to the other tools, is quite old. It is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence and has been around since 2014. Netflix open sourced it in 2015.
• Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Work on Argo CD started in early 2018, roughly one month after Jenkins X.
• Tekton Pipelines is the youngest of the tools being compared. It emerged from the Knative pipeline component at the end of August 2018 and describes itself as a tool that provides k8s-style resources for declaring CI/CD-style pipelines.
• Jenkins X was introduced in January 2018 and promises an open source opinionated way to do continuous delivery with Kubernetes, natively.

Right off the bat, we can identify Spinnaker as by far the oldest of this ensemble. The other tools did not exist before last year. All four tools share their open source licence. Jenkins X, Spinnaker and Tekton belong to the CD Foundation, a relatively new member in the long list of Linux Foundation Projects.

Not Covered in this Blog Post

Tools that we’re already using in production at inovex did not make it into this comparison. That includes the Gitlab CI components including pipelines and runners as well as the classic Jenkins, among others.

But enough with the introductory words. Let’s dive in!

Spinnaker: Mighty and Around for Ages

Let’s begin with the tool that has been around the longest: Spinnaker. It was originally developed at Netflix after they noticed Spinnakers predecessor Asgard could not keep up with the requirements for modern cloud deployment on AWS. The main focus of Spinnaker is to provide a cloud-agnostic way for deploying generic concepts like servers, server groups, load balancers and machine images to different providers. By utilizing a plugin concept for so-called cloud drivers, the development of Spinnaker features is decoupled from their implementation in plugins for e. g. Google Cloud, AWS, Azure Cloud, Openshift and more.

Deploy Wherever Your Teams Want

Because of this vast selection of different cloud providers inside Spinnaker, it can be used as a central tool for multiple teams. These teams can each have their preferences on which cloud provider they want to use and which steps their pipeline should consist of. Nevertheless, templates for pipelines containing company best practices can be managed and provided centrally. The configuration of all these parts for the end user happens trough Deck, Spinnakers UI component. This config can be saved in a cloud provider’s storage solution of your choice. Notice that this means there is no GitOps or infrastructure as code approach implemented in Spinnaker itself, even though pipelines can be triggered based on Git webhooks.

Using Halyard for Administration

The setup and administration of Spinnaker is done via a command line tool called “Halyard“. The Halyard Command Reference gives a first impression of the complexity one can achieve with a Spinnaker setup. In the context of Kubernetes-native CD, the cloud driver for Kubernetes comes into play. It allows for the direct monitoring and management of ReplicaSets and Pods in your destination cluster.

Spinnaker is not a Build Server

In a blog post about common misconceptions about Spinnaker, Kostis Kapelonis explains how Spinnaker uses third party build tools like Jenkins or Travis in order to not reinvent the wheel for CI. These plugins together with the component-based architecture of Spinnaker and its fine-grained configuration options make it extremely versatile, but also very complex to operate. Additionally, all training materials seem to heavily focus on the UI. The “Spin“ CLI tool doesn’t get promoted as much. We also mentioned the lack of Infrastructure as Code support. On the plus side, the documentation for Spinnaker is very extensive and mature. As an alternative to Spinnaker, Argo CD tries to fit right in. Let’s have a look.

Argo CD: a Lightweight Tool with Narrow Purpose

Argo CD is the next tool on our list. It is governed by the folks at Intuit (which you might know from products like QuickBooks or TurboTax) and belongs to the Argo Project, along with Argo Workflows and Argo Events. As a SaaS provider, Intuit actually used Spinnaker to deploy their software before switching to their very own Argo CD solution. They explained their reasons for this decision in a talk at KubeCon North America 2018. Long story short: Spinnaker was too slow and too heavyweight for them. As we’ve already seen, Spinnaker can be complex with all its cloud driver plugins for several providers and their respective VM/container products. If you only use Kubernetes, you neither need this kind of access nor should you have to care which cloud provider you’re using in the first place. Therefore, Argo CD is solely developed to work on Kubernetes.

The inner Workings of Argo CD

The basic principle of Argo CD is as simple as it gets. After you installed the tool to your cluster, you can manage your applications as follows: There are Kubernetes manifests that describe your app in a Git repository. The very same manifests are applied to your Kubernetes cluster, which satisfies the principle of infrastructure as code. Argo CD makes sure these manifests are always in sync. If a change gets merged to your master branch, the application is marked “OutOfSync“. Depending on your settings, Argo CD can automatically apply the changed manifests to publish the changes to your cluster. The option to manually trigger the rollout is also available, which allows for human interaction before rolling out to production. All the manifests and their sync state are presented in a clean, visually compelling UI, which Argo CD delivers out of the box. You can use both the UI and the Argo CD CLI tool to manage your applications with Argo CD.

Limits towards Continuous Integration

You may have noticed that the Argo CD approach does not involve any kind of CI, meaning there is no test or build step when syncing. This means that if you want a fully-fledged CI/CD pipeline using Argo CD, you need to combine it with other tools like Jenkins, Travis, Circle CI or Gitlab CI. You’d separate the application repository from your manifest repository. Changes to the app repo’s master would trigger your CI, testing and building the new container image. As a final step of CI, an automated commit would update the image version in the manifest repo, notifying Argo CD of the change. To summarize, this process could look something like this:

Niche Tool, Exciting Ecosystem

Argo CD does not satisfy DevOps professionals’ needs in that it’s not a complex tool to create highly customized pipelines containing lint, test, build and deploy steps. It focuses on the Git <-> Kubernetes sync control loop, which is an effective method for implementing infrastructure as code. The UI is great for visualizing the inner workings of a Kubernetes deployment, making it a useful option for onboarding devs that are new to Kubernetes. The Argo Project is an exciting ecosystem, with Argo Workflows being a pendant to Tekton Pipelines, which we’ll talk about next.

Tekton Pipelines: Small Footprint, Endless Possibilities

Tekton Pipelines is the youngest of the tools in our comparison. Like Argo CD, it focuses only on Kubernetes. The project originated in the Knative Build component and was outsourced into a separate repository in August 2018. The current version at the time of writing is 0.4.0, which shows how young the project is.

Kubernetes CRDs FTW

Installing Tekton Pipelines to your cluster introduces five new Custom Resources: Tasks, Pipelines, TaskRuns, PipelineRuns and PipelineResources. Tasks contain steps (which are essentially shell commands) and can be executed by TaskRuns. Pipelines consist of multiple Tasks and are executed by PipelineRuns. To share resources between different Tasks, PipelineResources can be used. Tekton just provides a framework for executing actions in a defined, acyclic way. Its power comes from the ability to execute whatever command in whatever container on a per-step basis. Steps inside one Task share a Pod for easy communication. With keywords like “runAfter“, you control precisely the order of the Tasks to run in.

Resources and Production Readiness

Currently, Tekton has four different kinds of PipelineResources: Git, Image, Storage and Cluster. These can be used to reference a code repository, a container image to either be built or loaded, to access stored files and to manage another Kubernetes cluster. The latter allows Tekton Pipelines to not only manipulate the Kubernetes cluster it was deployed to, but any cluster it gets handed as a resource.

A lot of the features of Tekton work great, considering it is still below a 1.0.0 version. Nonetheless, for non-trivial applications and to implement pipelines that have a more complex logic than just building one git repository and deploying the resulting image, a lot of work needs to be done by the developer. Inserting the desired container image with a command line tool into the YAML file as one task and deploying it without a commit back to the manifest repo in another task seems quite “hacky“.

Much more to Come: The Tekton Roadmap 2019

The Tekton maintainers plan to implement a long list of new features before the end of this year. This includes conditional execution, cancelling or pausing a workflow, resuming a paused or failed workflow and enforcing timeouts on Tasks and Pipelines. That such basic features are missing makes clear that Tekton, despite being used all over the place (for example in Jenkins X), is still a tool in alpha status.

Jenkins X: The More, the Merrier?

Finally, let’s have a look at Jenkins X. CloudBees is the driving force behind this elaborate approach towards a fully-fledged cloud-native CI/CD tool. To make it happen, the creators rely on tools that are already solving parts of the big puzzle that is cloud-native CI/CD. From Skaffold and Draft to Helm and Prow, every tool you probably heard of at the latest KubeCon has its place in Jenkins X. Don’t know what any of those do? Well, according to Jenkins X, you don’t need to.

Dissecting the Components of Jenkins X

The heart of Jenkins X is the jx command line tool which abstracts away the aforementioned tools. The first approach of Jenkins X was to use a containerized Jenkins inside a Kubernetes cluster to implement “cloud-native“ CI/CD pipelines. This makes it relatively easy to port legacy applications that come with a classic Jenkinsfile. However, this is not as cloud-native as we might like―there is still a JVM involved and we cannot utilize the resilience of distributed nodes as Jenkins runs as a single pod. To solve this, a second option is already built-in. It’s called “Serverless“ and uses Tekton Pipelines in combination with Prow. We already know Tekton Pipelines and how they work, so let’s have quick look at Prow.

This tool is so Kubernetes, it doesn’t even have its own repository yet. Prow is located in a subdirectory under github.com/kubernetes/test-infra/, and, spoiler alert, that’s what it does: This is Kubernetes’ very own CI tool, primarily responsible for testing and managing merge requests. As Prow consists of many microservices, the name game is stepped up once more. We have:

• hook, which listens to incoming GitHub webhooks,
• tide, which merges approved PRs,
• sinker, which cleans up resources that are no longer needed.

All these are running as containers in a Kubernetes cluster. The ProwJob CRD, which represents the life of a webhook trigger in the Prow ecosystem, is essential as well.

Putting the Pieces together

Now, how does it all come together? Let’s look at the following diagram:

Prow’s Hook component routes incoming webhooks from your GitHub repo to the PipelineOperator. This part is Jenkins X glue code to connect Prow and Tekton. It reads the jenkins-x.yaml file, which contains information on Pipelines, Tasks and PipelineRuns. You can think of it as an abstraction layer above Tekton Pipeline CRDs. The PipelineOperator creates the desired resources, which leads to application pods being deployed to the matching environment (staging, production or temporary PR environment). Crier reports back to GitHub and Tide merges the Pull Request as soon as all tests pass and a reviewer confirms the changes.

Problems with the Black Box

We didn’t even talk about the integration of Helm for chart management, Skaffold and Draft for a development experience in the cloud and the use of build packs for common applications. According to the creators of Jenkins X, you do not need to worry about what’s going on under the hood. For us, this seems to be only partially true. As all these tools themselves are still under rapid early development (e. g. Tekton is still alpha), bringing them together hidden under a blanket of CLI does not always work smoothly. And as soon as parts of the structure start to misbehave, you find yourself debugging tools you didn’t even know existed before. In a nutshell: Jenkins X is extremely powerful, but you still need to know a lot of the underlying tools and how they work in order to get a reasonable, sane understanding of what your CI/CD infrastructure is doing.

Cloud-native CI/CD: A Summary

As you can see, the cloud-native CD landscape is as versatile as the requirements can get. You have the heavyweight Spinnaker which combines multiple tools into a GUI-based approach to CD. Argo CD is lightweight, but doesn’t offer much in terms of flexibility: It syncs your git repo, and that’s pretty much it. On the other end of the spectrum, there is Tekton Pipelines, which allows for a creative approach as you can basically do everything you want in your pipelines. However, this also means there is a lot you have to figure out yourself. Jenkins X tries to minimize the “DIY“ part by providing an opinionated tool that is packed with other exciting projects that are already interesting on their own. Be aware of the complexity, though.

What do you think?

Is your favourite CD tool missing? Do you agree with our impressions on the presented tools? We’d love to hear what you think. Leave a comment! Or are you interested to hear more from us in a project of yours? Let us know, we’re excited to listen to your challenge.