Modern CI/CD with Jenkins 2 and GitLab CI [Comparison]

Continuous integration (CI) and continuous delivery (CD) are a great help, providing the flexibility needed for agile software development methods like Scrum and Kanban. With CI/CD, you don’t have to constantly struggle with the build and deployment processes of your software project. Once correctly configured, you can be assured that the whole build and delivery process is just a matter of pushing the code into the source code management system or even more simply pressing a button.

CI/CD in its current state

For a couple of years now, great progress was made in regards to continuous integration and delivery. Existing solutions constantly got better and lots of new products made their way into the industry. As a matter of fact, Jenkins just released its second major version earlier this year after more than ten years of development and maintenance of its first version. GitLab (with its integrated CI/CD module called GitLab CI) is relatively new to the game, however they are pushing new features on a nearly monthly basis. Besides Jenkins and GitLab CI, there are also some other wide-spread CI/CD applications like Travis CI, which is even free if you use it for open source projects, and GoCD which comes in the wake of Golang’s spread and rise in popularity, trying to be a big player in the CI/CD world, also.

Current situation at inovex

At the moment, the majority of our projects uses a self-hosted instance of GitLab EE as git repository manager. For continuous integration and delivery good ol‘ Jenkins version 1 is still up-and-running and in use. It has some disadvantages, though, like every project sharing the same slave(s) (Jenkins‘ runner machines) as well as its software and configuration. As an example, if one project needs software A and another project needs software B, but they are incompatible with each other, you have to find a workaround or instead use a separate slave. Even if you only update a toolchain, it might have disastrous consequences, because there might be other projects that depend on that specific toolchain version. It should be noted that some tools like Gradle or Maven can indeed have multiple versions installed without any conflicts. However, there are always tools which don’t support this.

As a result, we are looking forward to a state-of-the-art solution that doesn’t have these disadvantages. The following two options seem to be the most reasonable ones: Upgrading to Jenkins 2, because most of us are relatively familiar with Jenkins, or using GitLab’s integrated CI/CD tool ‚GitLab CI‘, because GitLab is already being used and its CI/CD module also comes with a lot of new and interesting features and advantages.

Similarities

Jenkins 2 and GitLab CI have a lot of things in common, and to not bore you to death while reading this blog post, there’ll just be a short explanation of the most important features that both CI/CD tools share.

Standard features that define the continuous integration and deployment process like automated builds, tests and deployments obviously are supported by both solutions. More precisely defined features like working in jobs, job chaining and manual triggering of jobs are supported as well. Additionally, there’s the possibility to use a central authentication provider and the CI/CD configurations are stored in the repository itself for both GitLab CI and Jenkins 2, assuming the usage of Multi-Branch Pipelines in Jenkins 2 (Please note: Building pipelines in the web UI is still possible). Things like having a clean and separate workspace are supported, even though they are implemented in slightly different ways.

Nice-to-have are features like the ability to execute jobs in parallel, automated code metrics and the possibility to cancel following up jobs if specific criteria are not met. Another wish was the easy creation of new build environments. For example, if a new version of nodeJS is released, you don’t want to upgrade to it immediately as this could have catastrophic consequences. Quickly being able to create another build environment using a separate Docker image and separate runner is much safer. Unfortunately, neither of them fulfill this wish, because of the necessity to write custom code (Groovy in case of Jenkins 2; Shell/Bash in case of GitLab CI) or setup a custom environment in Docker.

Differences

In the following sections, I will go over the most important differences of Jenkins 2 and GitLab CI. There will be a short explanation of what the feature consists of and if needed, a short example for clarification.

Configurability per Branch

The configurability per branch describes an option to either temporarily or permanently configure specific jobs for single branches (apart from the standard configuration which usually applies for all branches). With this option, it is possible that newly-created branches instantly use the defined jobs in the CI pipeline without any further configuration. Therefore, you can test new jobs or whole new workflows in their personal or feature branches without having the risk to affect already existing and working pipelines that are used for general branches like master.

With GitLab CI, the behavior is exactly as described above. Jenkins 2 only partially implements that behavior as it is not possible for individual jobs to differ per branch. Only the whole pipeline can be adjusted to only be triggered by specific branches.

Winner: GitLab CI

Time-based Execution

Sometimes it can be helpful to let jobs or whole pipelines be triggered based on time. For example, this is the case for regular nightly builds.

With Jenkins 2, this is possible out of the box. One can define in cron-like syntax the moment when jobs or pipelines should be executed. GitLab CI does not have this feature. However, it can be achieved with a workaround: A cronjob on the same or another server can be used to trigger jobs and pipelines by sending a web request to the integrated web API. Although this is not possible with GitLab CI, I personally think that most people won’t miss this feature. Why do you need nightly builds if the build from the last commit that was made at about 5 pm isn’t any different from the nightly build?

Winner: Jenkins 2

Automated Housekeeping

Housekeeping is important to prevent the collection of unneeded junk in the system, be it old build artifacts or just old build entries. It should be possible to set a time limit globally and per project or even pipeline when a build artifact or build entry expires, so it can be deleted. In addition, the amount of builds that will be kept per project should be configurable.

Unfortunately, neither GitLab CI nor Jenkins 2 have fully built-in support for this behavior. GitLab CI offers the option to define when build artifacts will be marked as expired and automatically deleted. However, this has to be configured before the actual build artifact gets uploaded and stored in GitLab as it has to be configured within the pipeline configuration. On the other hand, Jenkins 2 only allows to delete whole build entries (including their artifacts) or store a specific amount of them. However, like GitLab CI it is not possible to configure the default value globally.

Winner: I don’t know, you decide.

Pipeline History

In order to maintain a good overview of executed pipelines, there should be a function to sort and filter pipelines. Additionally, it should be possible to filter for already deleted branches.

Sadly again neither GitLab CI nor Jenkins 2 fully support this functionality as GitLab CI only shows the last pipeline of branches and both can’t show pipelines filtered for deleted branches.

Winner: Jenkins 2 slightly in favour

Dashboard

Having a dashboard for every project in a CI/CD environment is important, but what is even more important than just a dashboard is having a dashboard that also shows the information you actually need. You should be able to quickly scan the number of builds that succeeded or failed, when the last build succeeded or failed and preferably which parts of a pipeline succeeded or failed.

Jenkins 2 completely accomplishes that. GitLab CI lacks parts of the functionality described above. It only shows the latest build in the list of projects and for single projects it shows complete pipelines with all of their stages. However, GitLab CI does not display the specific jobs within the stages, so this can be a bit inexpressive.

Winner: Jenkins 2

Pull Request Support

An integration of the repository manager and CI/CD platform is very nice to have, so that you can see the current build status for pull requests. With such a feature you can avoid merging code into the main branches that does not work or isn’t even build correctly.

Jenkins has no further integration with source code management systems than just using it as is. GitLab and its CI platform are tightly integrated, so you can see running and finished pipelines for every open and closed pull request.

Winner: GitLab CI

Report Integration

If your CI/CD solution helps by directly reminding people of sudden changes in code quality, you can specifically tackle that issue without losing too much time. Therefore, reports should be preferably visible directly within the CI/CD environment and not on a separate page or even worse, as a downloadable report. Often, that’s too much hassle and too much time is wasted. I think if this information is not visible within a few seconds, you will never actually look at it more than once.

With GitLab, this is very limited as it is only possible to extract a specific part (with RegEx) from the console build/test output. In contrast, Jenkins 2 has the ability to display the build or test report directly on its pipeline page.

Winner: Jenkins 2

Visual History of Reports

As with our requirement for „Report Integration“, it should also be possible to view a visually appealing history of build and/or test reports to be able to see any kind of increase or decrease in code quality.

Jenkins 2 shows a small graph for the test report history directly on the project detail page. GitLab CI lacks this kind of feature completely.

Winner: Jenkins 2

Executor Support

Support for more than the standard shell execution pattern is crucial in a modern CI/CD environment, especially if you consider that an organization usually uses an automation server for more than just one project. As a result, multiple projects most likely have different requirements and dependencies when it comes to build environments. Huge incompatibility issues might occur if more than one version of a given tool is installed on the same machine. Therefore, other executors like Docker, Parallels or VirtualBox are a necessity.

Jenkins 2 has support for shell, and via plugins VirtualBox and Docker. In comparison, GitLab CI has out-of-the-box support for shell, SSH, Docker (including executors for some special edge use cases), Parallels and VirtualBox. The default option for GitLab CI is the Docker executor.

Winner: GitLab CI

Permission Inheritance (Repository Manager)

An inherited permission management from the repository manager is useful for larger groups of developers or organizations where you do not want to set the permissions for each user for each service (i.e. repository and CI/CD tool) individually. Most of the times the permissions are the same in both cases, so they should be configured in one place by default.

This feature is integrated into GitLab with its combined GitLab CI. They have an easy time here because of the deeply-integrated structure of those two. On the other hand, as Jenkins 2 does not come with a built-in repository manager, it does not have the ability to directly incorporate permissions between repository manager and CI/CD platform.

Winner: GitLab CI

Plugin Support

Plugin Support can come in handy very quickly. As soon as you find functionality that is either only partially implemented or even entirely missing, you can just search for plugins that cover your needs or you write them yourself. Especially considering that you will not switch your CI/CD application service every few months, it is quite useful to have the possibility to add more functionality to the existing environment.

Jenkins 2 has full support for plugins, written in the widely-spread Java language. In contrast, GitLab CI currently does not come with plugin support.

Winner: Jenkins 2

Two Factor Authentication

As simple as it sounds, just support two factor authentication for improved account security.

GitLab supports Two Factor Authentication while Jenkins 2 does not.

Winner: GitLab CI

Conclusion

The CI/CD solutions GitLab CI and Jenkins 2 both were developed to fulfil exactly the same need: automating processes for continuous integration and delivery. Apart from some similarities, there are quite a lot of differences between the two contenders. GitLab CI is a fixed component of the Git repository manager GitLab and therefore offers great interaction between CI/CD processes and repository functionality. Jenkins 2 on the other side is loosely-coupled from any repository manager and as a consequence it is very flexible when it comes to the selection of version control systems. Furthermore, just like its predecessor, Jenkins 2 has an emphasis on plugin support to further extend or improve existing functionality of the software.

In addition there are substantial differences in regards to general overview of projects or builds. For example, the dashboard in Jenkins 2 is way more flexible and adaptive with its ability to integrate build and test reports into a general overview of projects or builds. Here, Jenkins 2 is miles ahead of GitLab CI.

However, there aren’t any world-shattering differences between the two and considering the requirements posed, Jenkins 2 barely wins the race. In Practice the difference is marginal, so one might also consider it a fair draw, even though this is technically not true.

Finally, as the writer of this blog post, I have the opinion that GitLab CI has the greater potential because of the fixed integration into the git repository manager GitLab. Features like permission inheritance and two factor authentication are highly appreciated in enterprise environments. Almost certainly, developers will add important (comfort) features like the ones that already exist in Jenkins and other CI/CD platforms to further improve the practicability of GitLab and GitLab CI. Currently, it is a draw between Jenkins and GitLab CI, however, sooner or later GitLab (CI) might have the same amount or probably even more features than Jenkins and other CI/CD services.

In conclusion, we at inovex have chosen to use GitLab CI as our default tool. For projects with special needs and requirements, the project teams still have the choice to set up a different CI/CD solution (even the legacy version of Jenkins, if they need to. However, that won’t hopefully happen too often 😉 ).

Read on

Have a look at our website to find out more about the services we offer in IT Engineering, write an email to info@inovex.de for more information or call +49 721 619 021-0.

For an updated comparison of cloud-native CI/CD tools have a look at our latest article.

Join us!

Looking for a job where you can work with cutting edge technology on a daily basis? We’re currently hiring Linux Systems Engineers (m/w/d) in Munich, Cologne and Hamburg!