The Top 7 Things That Your CI Server Should Do

As I mentioned in a previous post, we are using Jenkins CI.  Here are the top 7 things that we are using it for.

1. Build from Source Control

This is the most basic, but also the most crucial.  If you don't know what source code is being used in production, then you are in some trouble.  If your CI server can build it, then you guarantee that at least one other person can build the code.  If not, then you don't know if someone has some magic settings in their IDE or in their environment variables.  Does it guarantee that you can build the code on your machine?  No, but it will give you a better chance.


Your CI server should pull (check-out) your code from your source control system, compile your code, and create the artifacts that are needed for deployment.  I would highly recommend using a build tool like CMake (C++) or Gradle (Java) to help facilitate building.  Gradle has some amazing dependency management built-in and a lot of plugins that help.  Additionally, adding a build script in your source control system provides trace-ability.  It allows the user can go back to the exact point in time when you created the build to recreate the artifacts if needed.  If you don't know the build parameters, then it gives you less chance to recreate the artifacts.

2. Test + Test Coverage

Running all those tests you created are crucial.  We automatically fail the build if tests fail, but that is really up to you.  Also, it can be hard to figure out how many tests you should run when doing a basic build.  Should you run integration tests and regression tests?  We have found that keeping normal builds, the ones that run after a push (check-in), to under 2 minutes or so is helpful.  As in website usability, quick feedback is essential.  For those long running tests or suite of regression tests, we have those run overnight in a nightly job and make sure they run during a release.

It is also nice to keep track of changes of code coverage as new commits are added.  Jenkins has two great plugins for this: Emma and Cobertura.

3. Track Compiler Warnings

As you build, track your compiler warnings.  When we first setup Jenkins, we would break the build when people pushed up code that had warnings in it.  We were very strict.  

After attending the GOTO 2013 conference in Chicago (a great conference by the way), and spending the day with Michael Feathers (author of Working Effectively with Legacy Code) he recommended that you keep your build gates low.  In other words, promote people to commit and push code very often.  If you make it difficult for people to and push code to the central server, then they will do it less often.  If they push code less often, you spend more time integrating each other's code and there is less collaboration.

Even if you don't break the build though, tracking these is important.

4. Static Analysis + Scan for TODOs

Use the static analysis tools for your development stack.  For Java we use PMD (primarily) and for C++ we use both cppcheck and PC-lint.  In general, I would go through the rules and exclude the ones that you don't see important.  For example, for PMD it seems like if you have a variable name that is less than 5 or greater than 15 characters it gets flagged as too long or too short.  If you really care about that sort of thing, then definitely use it, but otherwise exclude those rules that just don't make sense.  These are guides to help you code better and find potential bugs before they wake you up in the middle of the night.  They aren't the bible.  

Additionally, depending on team and codebase, it might be good to track TODOs.  We also inherited a ton of legacy code that we didn't know a lot about, but we knew it contained a lot of TODOs.  When we first started tackling refactoring, it was nice to watch the number of TODOs diminish to 0.

5. Memory Leak checking

How do you know if you have introduced a memory leak?  You setup a valgrind job to test your C++ code.  This saved us.  When we first tested our legacy code with valgrind, we found a few memory issues and fixed them.  Later, a memory leak popped up that we would have never caught if we didn't setup the valgrind job.  We haven't setup anything yet for Java, but I would be open to the idea.  If you have suggestions, please comment.

6. Performance Profiling

Knuth says: "premature optimization is the root of all evil," but I think many software engineers take this as "optimization is dumb."  I find very few developers who profile their code...ever. How do you know if you need to optimize if you don't measure your performance and track the performance of your application over time?  You could check in code and double runtime of a particular task, but would you even know it happened?  I was a little disappointed that the first time I saw a profiler was in grad school (in Advanced Graphics).  My point is just that you should measure performance, and then maybe optimize your performance when a new change is introduced and something doesn't quite look right.

Track your test times for regression and integration types of tests.  Use timeouts to warn when things aren't going as fast as you would like.

We also used gprof with C++ code to measure performance.  Then we used gprof2dot to generate a call graph.  This proved beneficial on legacy code.  We quickly found a function that was called many more times than it should have been.  We noticed that the function was called in two different locations in the code.  The function was a complex calculation so we ended up caching this information and only calling it once.  For a while, we had the call graph job ran every night to keep track of things.

7. Release Management

We utilize the release plugin to handle releases.  We also use the parameterized build plugin to keep track of versions.  Our release flow goes something like this:

1. We pass in the changset and branch that we are releasing
2. We have pre-build (release) step that switches our mercurial repo to that specific changeset.
3. We perform the normal build steps
4. If the normal build is successful, then we tag the changeset with something like "Release 1.2.345" and push that tag back to the server.

Setup deploy jobs that facilitate basic deploying.  If you have a war file that gets generated, and it needs pushed to a jboss/tomcat/glassfish server when you are ready to release, then write some automation to let the proper artifacts get copy/deployed to the correct servers. 

I wrote a simple gradle jboss deploy plugin that helps us deploy using gradle.  In Jenkins, we just need to point to that gradle task and run something like:  gradle deploy

If you have questions or comments, please leave them.  It would be cool to see how others are using Jenkins or other continuous integration servers.