There are several dozen CI Build tools available in the industry. I recently did some research on some of the tools and this is what I found. This is not a comprehensive list but a good start to I hope a heated discussion about the benefits of these tools. The tools I evaluated are:
- Bamboo - Atlassian
- TeamCity - JetBrains
- QuickBuild (Formally LuntBuild) - PMease
- Jenkins (Formally Hudson) - Open Source
CI Build Benefits
I found that these four tools have many of the same benefits such as:- Integration
- Defect Management Tools - Jira, Bugzilla, Redmine, TeamForge, Trac
- SCM Tools - Subversion, CVS, Perforce, ClearCase, Git, TFS, etc…
- Features
- Pre-Commit Test and Continuous Integration
- Extendable through REST API
- Supports ( Ant, Maven, MSBuild, Nant, Rake, …)
- Security Access Control
- Build Notification and System Alerts
- Metrics Aggregation
- Custom User Dashboards
- Build Setup and Workflow Design
- Integration with Static Analysis
- Test Insight and Statistics
- Advanced Build Grid and Cloud Integration
- Build Pipelining and Continuous Deployment
CI Build Tool Differentiation
All of these tools have these features that in common and I would say these are necessary in order to have a CI Build System. Without these they would not be feasible for the type of software development work.So there has to be something that differentiates the different tools from each other. I found three key differentiators in these tools.
- Build and Workflow Definition
- Machine Health and Monitoring
- SCM Integration
Build and Workflow Definition
Build and Workflow Definition show how easy it is to define build steps, dependencies between builds, and aggregate build definitions.Bamboo * * *
Bamboo has the concept of Phases, Stages, Jobs and Tasks.Stages can be run sequentially or in parallel and artifacts are passed between stages. Stages are made up of several jobs that are run in parallel. Jobs are made up of tasks. Tasks are commands that are run on Build Agents. Tasks are run sequentially. Bamboo focuses on a very Release Management tops down approach.
Artifacts are essential to passing data between stages of the build. Bamboo is built with a multi-tiered/multi-component pattern in mind. Where products are made of releases from multiple component releases and tested together.
Jenkins *
Jenkins is one of the older tools out there and is in the open source community. It came from the Java world and is still very Java centric. Most jobs are commands and/or maven targets. This works great for simple builds that run command and/or tests. There is a Multi-job plugin for Jenkins but it is not as powerful as the other tools.TeamCity * * * * *
Looking at TeamCity it appears that it was built around a great workflow engine. You can tell it was built from the ground up with complex workflow and simple workflows in mind. They have the concept of Hierarchical Projects, Job Configurations, Parameterized Jobs, and a cool concept call Meta-runners that let you reuse parts of your build in other builds. They have both Screen driven workflow definitions as well as XML text based definitions. Artifacts are part of their system but not to the same degree as Bamboo. This tool ranks the highest in Workflow and Build Definition.QuickBuild * * * *
QuickBuild takes a close second to TeamCity for Workflow and Build Definition in my book. They have many of the same concepts of TeamCity but does not take it to the same level. They have Hierarchical Projects, Jobs, and Job Templates. They do not have the Meta-runner concept that TeamCity has which is a small negative for them. But they share many of the same types of Screens and Workflow Editors that are found in TeamCity. Artifacts are part of the their system but again they are not as good as Bamboo in this respect. This tool ranks second right behind TeamCity in this category.SCM Integration
All of these systems do a good job with integrating with SCM triggers for changes, displaying change lists for each build and polling the SCM systems for changes. But there are some differences that make one of the tools standout.Bamboo * * * *
Bamboo has the best integration with the SCM systems especially Git. Bamboo was built around the concept of moving things from one stage to another and promoting build artifacts and their code to different branches in the SCM system. This concept works really well to keep large team isolated from build breaks of components that are still trashing with change. They also have the tightest integration with Jira with automated Release Notes, Build Break Reports, Recurring Issue Build reports etc... Very well done for Bamboo on it integration into the SCM system for this they get the highest ranking.Jenkins * *
Jenkins has good integration with the SCM systems and allows for quick build trigger definition when changes happen in the SCM system. They also do a good job showing changes for each build. And surprising for the simpleness of the tool they have build promotion built into many of the default jobs. Most of these are very Java and Artifactory centric, but they do get the job done. It is not the top in this category, but it gets 2 stars for having everything necessary and a couple of extras.TeamCity * * * *
TeamCity has the standard tools needed for integration into the SCM system but it does not have the integration with branch and merge management that Bamboo has. You can define the jobs yourself with their robust job definition tools, but it is not first class like Bamboo. They do have a good interface to Defect managment tools and to build break reporting. They have recently introduced Remote run and Pre-Test Commits to their tool box to help cover many of the holes that organizations have in their build infrastructure. That helps but it still is not at the level as Bamboo for their integrations. So I am giving it 4 stars. Right behind Bamboo.QuickBuild * * * *
QuickBuild and TeamCity are equally matched in the integration they have with SCM and Defect managment tools. QuickBuild has all of the normal trigger based and polling SCM systems for changes and listing Change sets like all of the other tools. They still don't have the tight integration to branching and build promotion that Bamboo uses, but they have their own unique Build Promotion concepts that give it an advantage over Jenkins. Because you can make QuickBuild do everything that Bamboo can do with its build and branch management through the Build Definitions, I am giving QuickBuild 4 stars to match that of TeamCity.Machine and Resource Health and Montiroing
Bamboo - * * * *
In the last 4 years bamboo has done a great job at integrating with Amazon's EC2. The integration allows users to define Capabilities of EC2 images and then define their jobs that use the capabilities. This concept reminds me of the old Distributed resource management days with tools like Condor and PBS. Mapping supply and demand to run jobs. They have done this with a simple to use interface. You will however need to get on your EC2 Ninja belt to make sure that your EC2 image actually has the capability. [:)] Build agent reports are available for viewing, but the tools and reports are not at the level of a grid technology solution. For this reason I am giving Bamboo 4 stars.Jenkins - *
Jenkins allows you to have multiple build agents, but the agents are tied to specific jobs/builds. Basically a poor mans server farm. But its simplicity is its strength. It gets the job done for small teams, and small builds. Which for most projects is just right. Because it does not have a dynamic resource pool of machines I am giving Jenkins 1 star.TeamCity - * * *
Team City recently added an Amazon EC2 extension to their library of tools. This extension is still in the beginning and does not have all of the features that Bamboo or QuickBuild has. TeamCity has the concept of pools of build agents or machines that are assigned to projects, builds, and/or steps. This allows for different teams to be using the same machines with different priorities on builds in the machine pools. Because TeamCity does not have has much as the dynamic tools I would like to see I am ranking right below Bamboo and giving it 3 stars.QuickBuild * * * * *
QuickBuild looks like it was built with Grid Technology in mind. The tools and screens that manage the machine resources are great and you can quickly see what is going on. They have the concept of mapping resources and jobs together. Jobs have a set of resources that they consume and machines have resources to be consumed. They have an integration with Amazon EC2 much the same way as Bamboo. Because Quickbuild was built with multi-platform/OS builds and tests in mind they get 5 stars for this category.Summary of findings
After all of this I don't have a good answer to what tool is best. They have great benefits that cannot be overlooked. As with many development tools it is based on the work that you are doing. So here is the summary of the tools and their strengths.Bamboo
- Good for Component team fitting in a bigger group.
- Learning curve just above Jenkins, but not much.
- Simple to get up and running and easy to create more complex jobs
- SCM/Release/Deployment centric.
- Great integration with EC2 and VMWare
Jenkins
- Good for a small group.
- Simple to get up and running.
- Free for download
TeamCity
- Learning curve higher than Jenkins similar to Bamboo
- Job and Project Definition the easiest to understand and get up and working.
- Most Build Definition Centric tool
QuickBuild
- Learning curve is higher than Jenkins or Bamboo
- Very Grid Technology centric. Build Server Farms
- Great for very large and complex builds and teams.
- Very flexible and extendable
DWP
