• Continuous Integration eats itself (Part IV of SQA tools series)

    by  • March 16, 2015 • Series of SaaS Tools for Software Development • 0 Comments

    The series of SaaS tools for software development continues this time with Continuous Integration tools. As you can see in the previous posts in this series (on ALM and Test Management tools) we are only focusing on those tools that are available in the cloud, also called SaaS, Software as a Service. This time we are very interested on how they monetize their platforms.

    Continuous Integration Tools

    Continuous Integration is another of the concepts highlighted by agile development methods. Agile, and specially lean, methodologies aim for quick iterations of product releases or deployments. Continuous Integration tool check that the recent code changes can be quickly deployed to the actual product without breaking anything. This goes hand-in-hand with the availability of libraries and services that do automatic testing.

    Companies usually chose between running the test and builds in-house or delegate in some cloud solution. However, the elastic demand for testing and building makes CI a good choice for a combination of PaaS and SaaS. Companies can switch on machines instances in the cloud (e.g. Amazon VM) on demand, or just get a SaaS provider to run the build and tests on demand (e.g., new code is committed to the master branch). As well, hybrid solutions exists, where the SaaS provider will launch new VMs in your name (so that you pay for them).

    Logos of continuous integration tools : travis codeship circleci jenkins bamboo solanolabs snap semaphore droneio appveyor

    Logos of continuous integration tools

    SaaS tools

    Appveyor

    Appveyor is a CI solution that is trying to differentiate itself by adding support to Microsoft development platform, including MS SaaS solution Visual Studio Online.

    Appveyor commercial pricing starts with $60/month for one concurrent job. Extra concurrent jobs will add $40/month per job.

    Atlassian’s Bamboo

    Atlassian has been little by little cornering the market with tools that nicely integrate with their Jira platform. Bamboo is a very complete solution that also plays nice with other tools.

    As usual with Atlassian, you can find a free trial and the starter package ($10/month) to hook in start-ups. The starter package is interesting because it limits the number of jobs that can be defined in the system to 10. A job is basically a single task (e.g., run make command), and they can be organized in Plans, where you can easily chain jobs.

    Normal package provide unlimited jobs, and agents that execute the plans. The first agent is $50/month and price goes down as you “hire” more of them to work concurrently (5 agents go for $100/month). However, the catch is that the user/company has to provide the keys to their Amazon EC2 account, as it is where the agent will be run. Amazon will then bill you/the company for the time the EC2 machines are up.

    CircleCI

    One of CircleCI strengths is its integration with GitHub, and at the same time it is the biggest drawback, as it lacks support for any other SaaS providers of source control (e.g.,Atlassian’s bitbucket). Also, CircleCI can automatically divide your test and run them parallely, making the integration processes significantly faster.

    As for pricing, you can get the first container to run your tasks for free. However, to take advantage of the parallel execution of tasks and tests you will need more than one. Each extra container cost $50/month.

    CloudBees’ Jenkins

    Jenkins has gained lots of popularity thanks to its open source approach. You can run it for free in your own servers, or improve it. One of the main advantages of Jenkins is that it has lots of plugins, so it will adapt easily to any company development toolset.

    Cloudbees, the company behind Enterprise Jenkins offers a SaaS solution for those who do not want to take that route. Due to the open source nature of the software, the pricing options are more creative and two factors determine the price to pay. The main factor is the number of users of the platform. The second factor is how long your tasks take in the Cloudbees server, as you will be billed by the minute of computer time.

    The starter package includes 5 users, 5GB of space for $60/month. All of the packages include parallel builds and the price per minute is the same: $0.425/hour (standard speed) or $1.32/hour (high speed).

    Codeship

    Codeship main differentiation ingredient is the concept of parallel test pipelines, Test pipelines are automatically populated with tests and run in parallel virtual machines, making the build process much faster.

    Codeship has a free plan, where you are allowed to 100 builds per month with one concurrent build and one test pipeline, aka no concurrency at all. For $50/month you get unlimited builds, and one extra parallel test pipeline. More concurrent builds come for $150/month, which gives you 3 concurrent builds.

    Drone.io

    Drone.io is proud about how flexible your build environment can be. That feature is specially useful when you have complicated development requirements that do not fit exactly into the settings of a plugin or extension.

    Drone.io adds the number of projects as a pricing factor. However, the number of concurrent builds s the main price factor. The starter package gives you one concurrent plugin and 5 projects for $20/month. Four concurrent builds will cost $200/month.

    Semaphore

    Semaphore excels in the easy configuration of build and deployment tasks. Semaphore finds automatically the tasks in your source code and lets you divide them in different thread.

    The free package gives you a lot of power, it includes 2 processors. But it limits what you can do with it as only 100 builds are allowed per month. $30/month gets you unlimited builds and 2 processors. More processors come at about $50/month per processor.

    Snap CI

    Snap CI is meant for the web, as it supports almost all of the web stack, popular languages, databases, deployment solutions and, importantly web-testing frameworks. However, they only seem to support github as a source origin.

    Their free package is only meant for public repositories. The small team will give you two concurrent builds for $80/month.

    SolanoLabs’ SolanoCI

    SolanoLabs’ CI solution is oriented to bigger development companies that have lots of tasks running parallely. Thus, that cannot spend time deciding how many workers/processors they will need (most likely a lot of workers). SolanoCI markets itself on being fast. As in Cloudbees, you pay for the time the processors are working for you.

    The individual starter package gets you 2 workers and 10 worker hours for $15/month. Additional hour will cost $0.75. Going pro starts at $125/month. If you are hosting your app at heroku. SolanoLabs offer their Pro package for $150/month for 8 workers and 256 worker hours.

    Travis CI

    One of the big names in continuous integration come from Berlin. TravisCI started popularizing the CI concept by making it super easy to add the capability to GitHub projects. Now it is considered the standard for open source projects, and its dashboard at travis-ci.org gives a quick view of how popular, and not so popular software gets built.

    The Travis-CI SaaS solution for private projects have a pricier starter pack. The startup package gets two concurrent jobs for $130/month. Thought, if you require more jobs up the price per job stabilises at about 50 per concurrent job.

    Summary

    The need for continuous development only increases and mature companies should just keep getting better at it. IBM has a great document on how to improve at continuous delivery that you should read.

    The SaaS market analysis reveals the popularity of GitHub. The reasons are varied. Most, if not all, offer a free package for open source, and GitHub is where open source happens nowadays, goodbye GoogleCode! Thus, attracting those developers is important before moving them to the paying packages. This allows for niche players to deliver good solutions for other stacks. For example, Appveyor caters the Microsoft development stack, and Snap-CI aims to build a better CI for web-apps.

    The three big competitors attract different crowds. Bamboo relies on people using the Atlassian solutions. Cloudbees builds on the open-source credit of Jenkins, and Travis-CI on the popularity of its free tier in the GitHub community. However, new competitors keep bringing new solutions, specially focused on the performance of build tasks, specially by using parallel execution of tasks.

    On one side, most of the SaaS vendors price their offering depending on the amount of processors/workers/builders available to the user. The price of those processors is about $40-$50 dollars in most of the platforms. On the otherside, Cloudbees and SolanoLabs have decided to bill per minute of execution. So platforms compete also in business models.

    For other comparisons of SaaS CI tools you can check the user-curated one from Slant, and the post from Strongloop.

    About

    We are a strategy consultancy. We work with companies & institutions worldwide on strategy crafting, internationalisation & business modelling.

    http://www.vaibmu.com