How I learned to stop worrying and love the CI server

Kategori: Dev

NRK TV and its API has been through a transformation from a monolith to a service-orientated architecture (notice how I avoided the word μικρόservices). We are dogfooding our API from as if we were a smart-TV or a mobile app. We have also moved from TFS to Git.

With smaller components change has a limited scope. With Git we can test new ideas on different branches without being afraid of breaking things. Not being afraid is important to healthy maintenance and rapid deployment of new features (I leave it to the reader to figure out which word I just avoided).  We can easily test new code on separate branches, and our tests will often catch us if we break something. But there is one piece left that we avoid making changes to:

The scary build server

Changing configuration on the CI server requires a certain panache.
Changing configuration on the CI server requires panache, Foto:

The dependency graph is the scariest part.

«You see, we can’t build on your agents because you have Azure SDK 2.7 but we are still on 2.6 and, and you are on XUnit 2.1.0 but it doesn’t work with dotCover as it is bundled with TeamCity and XUnit 2.1.0 needs an upgrade of NuGet.exe. Also, you see we have installed the karma runner globally with node.js and now we aren’t compatible accross projects.»

If you are like me, hearing such a conversation makes you realize something have to change. What becomes obvious is that we need the capability to run builds with different configurations on different branches of the build. We need the build configurations under source control and a tool that can run independent of TeamCity.

Configuring the build in a proper programming language would be preferable. For the front-end devs that work with js all day long, grunt or gulp or similar tools seem like natural choices. I do pity them for having to do advanced configurations in js, but they seem to be happy and their build pipeline works well so I am not going to interfere much.

For the .NET parts we have more choices, psake, Cake, FAKE etc. I will dismiss using PowerShell with very little discussion, and we do need something that has support for our ecosystem, such as OctoPack, Azure and TeamCity (yes, we still need it!). Cake is C# based (, and seems like a natural choice for C# developers. FAKE is F# based, but the DSL part of FAKE is quite easy to learn. We chose to use FAKE for its better typing and more concise build definitions. The idea is that the powerful build DSL should be reward enough for anyone willing to learn a little F#.

A migration plan, sort of

Where do you start when migrating a solution with 48 Projects and more than 180 nuget References that builds as one massive CI build using Visual Studio solution files?

We chose to move one and one project build definition from TeamCity configuration to FAKE, leaving the original builds running in parallell until we could confirm that everything was stable. When a part of the original build is migrated to FAKE and considered to be stable, we simply configure VCS settings in TeamCity to not trigger on changes to those parts. In principle we build mostly two websites and a few backend-services, and we have now split the two websites out to build purely on FAKE. All TeamCity does is trigger a build.cmd file.

The simplest thing was to take the API that does not have any js parts first. After that we moved the web frontend to a separate build, using the brand new NPM features of FAKE.

Building locally. A flashing console makes you look good in an open office.

Now I’m not afraid of the build-server anymore.

The fantastic thing now is that everyone can easily make a branch, in the workshop, push it to TeamCity and try out different experiments without breaking anything for anyone. If it works, we merge, if it doesn’t work, you haven’t caused issues for anyone. And if you have to troubleshoot it, you can try it out locally on your own machine.

The next step is to make sure no-one else are afraid either, and we will do that through an internal FAKE workshop, teaching just enough F# to be able to modify and run build-scripts and a little bit of the !!, ++ and — syntax in FAKE.

To show a real, messy build-script I included a raw one from our current build definition. It’s not perfect yet, but now no-one needs to be scared of refactoring build definitions and make a beautiful build of it!

This is the report from building locally. Beautiful.

6 kommentarer

  1. Nice write up of how things should be done :). I also get the feeling the the build is faster when running them with FAKE (or some other build script) compared to running everything on server, is that something you’ve noticed too?

    So the NPM part I submitted last week was the last piece of the puzzle? 🙂

    Svar på denne kommentaren

    • Bjørn Einar Bjartnes (NRK) (svar til Tomas)

      Absolutely. Your PR was critical to be able to trigger all the javascript stuff that is going on – it is a lot going on with grunt and karma and what-not! Probably worth a blogpost of its own how we use that Npm runner, and also how we are not using global node.js versions anymore, but installing specific versions of node and npm using nuget.

      Regarding speed, the major speedup is that it is now easier for us to split builds in separate configs, so while a main-solution build took 13 minutes, we can now build the main builds than run often at 5 and 7 minutes each. Some things are actually slower, though, it is faster to use a global node.js version than installing a new one using nuget, even if it is cached… But it it much faster to spin up new build agents (as we are doing now), and much faster when we actually need to upgrade or run side-by-side with projects using a different npm-version (such as maybe npm 3 – it is coming soon…)

      We did end up having to submit a few PRs ourselves as well to get XUnit and MSTest work with DotCover the way we wanted. Turned out the Fake source code was quit easy to work with.

    • Bjørn Einar Bjartnes (svar til Tomas)

      Not yet, but planning to do it. Want to move everything to FAKE first, still some things that aren’t migrated.

Legg igjen en kommentar til Tomas Avbryt svar

Din e-postadresse vil ikke bli publisert. Obligatoriske felt er merket med *. Les vår personvernserklæring for informasjon om hvilke data vi lagrer om deg som kommenterer.