How I learned to stop worrying and love the CI server

NRK TV and its API has been through a transformation from a monolith to a service-orientated architecture (notice how I avoided the word μικρόservices). We are dogfooding our API from tv.nrk.no as if we were a smart-TV or a mobile app. We have also moved from TFS to Git.

With smaller components change has a limited scope. With Git we can test new ideas on different branches without being afraid of breaking things. Not being afraid is important to healthy maintenance and rapid deployment of new features (I leave it to the reader to figure out which word I just avoided).  We can easily test new code on separate branches, and our tests will often catch us if we break something. But there is one piece left that we avoid making changes to:

The scary build server

The dependency graph is the scariest part.

«You see, we can’t build on your agents because you have Azure SDK 2.7 but we are still on 2.6 and, and you are on XUnit 2.1.0 but it doesn’t work with dotCover as it is bundled with TeamCity and XUnit 2.1.0 needs an upgrade of NuGet.exe. Also, you see we have installed the karma runner globally with node.js and now we aren’t compatible accross projects.»

If you are like me, hearing such a conversation makes you realize something have to change. What becomes obvious is that we need the capability to run builds with different configurations on different branches of the build. We need the build configurations under source control and a tool that can run independent of TeamCity.

Configuring the build in a proper programming language would be preferable. For the front-end devs that work with js all day long, grunt or gulp or similar tools seem like natural choices. I do pity them for having to do advanced configurations in js, but they seem to be happy and their build pipeline works well so I am not going to interfere much.

For the .NET parts we have more choices, psake, Cake, FAKE etc. I will dismiss using PowerShell with very little discussion, and we do need something that has support for our ecosystem, such as OctoPack, Azure and TeamCity (yes, we still need it!). Cake is C# based (http://cakebuild.net), and seems like a natural choice for C# developers. FAKE is F# based, but the DSL part of FAKE is quite easy to learn. We chose to use FAKE for its better typing and more concise build definitions. The idea is that the powerful build DSL should be reward enough for anyone willing to learn a little F#.

A migration plan, sort of

Where do you start when migrating a solution with 48 Projects and more than 180 nuget References that builds as one massive CI build using Visual Studio solution files?

We chose to move one and one project build definition from TeamCity configuration to FAKE, leaving the original builds running in parallell until we could confirm that everything was stable. When a part of the original build is migrated to FAKE and considered to be stable, we simply configure VCS settings in TeamCity to not trigger on changes to those parts. In principle we build mostly two websites and a few backend-services, and we have now split the two websites out to build purely on FAKE. All TeamCity does is trigger a build.cmd file.

The simplest thing was to take the API that does not have any js parts first. After that we moved the web frontend to a separate build, using the brand new NPM features of FAKE.

Now I’m not afraid of the build-server anymore.

The fantastic thing now is that everyone can easily make a branch, in the workshop, push it to TeamCity and try out different experiments without breaking anything for anyone. If it works, we merge, if it doesn’t work, you haven’t caused issues for anyone. And if you have to troubleshoot it, you can try it out locally on your own machine.

The next step is to make sure no-one else are afraid either, and we will do that through an internal FAKE workshop, teaching just enough F# to be able to modify and run build-scripts and a little bit of the !!, ++ and — syntax in FAKE.

To show a real, messy build-script I included a raw one from our current build definition. It’s not perfect yet, but now no-one needs to be scared of refactoring build definitions and make a beautiful build of it!

This is the report from building locally. Beautiful.