nrk.no

Undoing the harm of layers

Kategorier: Dev,English-articles & NRKTV

Partitioning the big ball of mud

As we approach a new problem we look for strategies to decompose that problem. The primary axis we choose for decomposition is fundamental to the architecture of the system we build. Technological boundaries are easy to identify in the codebase, and as such it is a tempting decomposition strategy. Strategies such as domain-driven design argue for boundaries that require more thinking and design. I will illustrate the harmful strategy of partitioning by technological boundaries by using static analysis on a real life system. I will show that a simpler design was achieved using a domain-based decomposition strategy.

As an example I will use the system I have been working on for the last couple of years, the API for NRK TV, NRK Super and NRK Radio. It serves web-sites, smart TVs and mobile phones with information about programs, recommendations, electronic program guides, personalized content, search etc. It has developed from primarily being the the backend code for a website, to become an API serving multiple clients. A lot of functionality has been added over the years. More than ten backend developers have been involved, half of which are not currently on the team.

A while back we started to reimplement parts of the system using a different decomposition strategy, and new functionality has been implemented in the new system. The playback functionality is among the most complicated parts of the system, and as it was reimplemented in the new API, it serves as a good basis for comparing the two strategies and their resulting complexity.

Our legacy partitioning strategy: Technology oriented decomposition

The signs of this strategy are easy to spot when you approach a new codebase: you will not find domain concepts before you look deeper into the code. It is like the developer wanted to hide the domain from you and present the problem from a clean, technical perspective.

At a code-level everything looks neat. You can be sure you will not get any of that yucky domain on your hands by browsing this codebase!

Assembly and namespace dependencies in legacy codebase

At assembly level, everything still looks tidy. Your primary axis of decomposition is almost always possible to get right. Adding new concepts to these boxes is easy, it is so generic almost anything fits inside. We must go deeper to see what the negative effects are.

NDepend assembly analysis of legacy architecture

Now, let us look one level below the assemblies. Let us open up all the namespaces and their dependencies. It turns out everything is just utter chaos. Technological layers implies that every piece of functionality is connected through shared assemblies and common namespaces. This is why I argue decomposing a problem by technology leads to the well known ball-of-mud architecture. In C#, due to the possibility of mutually dependent namespaces, the results are horrifying. It is only through well-disciplined testing and adding small pieces of functionality at a time, we are actually able to ship this as a running system.

One level below assemblies the real architecture shows

Focusing on a domain concept

If we focus on only one domain concept, playback of given program, it is really hard to spot where in the codebase only playback is involved. Since the playback controllers lives with all the other controllers, it becomes harder to separate which dependencies belong to only playback controllers from the other controller’s dependencies. I plot all the namespaces that the controllers depend on. A little bit of reduction in complexity is observed, but still not something you can wrap your head around.

All namespaces involved in playback, I still struggle to get my head around this

Technical debt

Out-of-the box ndepend rules indicate that we pay an annual interest of 54 days each year due to this complexity. More information on the calculations can be found here, we have currently not looked much into these numbers. Even if we do not use the numbers directly, I find them interesting to use when comparing technincal quality.

Out of the box analysis of legacy code

Matrix view of namespaces

Due to the fact that the graph is cyclic, it is slightly unfair to present it as a graph. Graph-visualization tools tend to give up making a neat layout. The matrix shown below gives a slightly more accurate description of where cycles between namespaces exists in the codebase. For more information on how to read the matrix see the NDepend docs.

This is not the javascript equality table, but it shows dependencies between our namespaces. Green and blue illustrates the direction of dependencies, black equals dependencies both ways. Namespaces with red boxes illustrates a set of mutually dependent namespaces.

Our new partitioning strategy: Functional oriented decomposition

Boundaries are not as clear when it comes to a functional oriented decomposition. Strategies such as Domain Driven Design can help us, but ultimately it requires domain knowledge and thinking to define suitable boundaries.

Strategies for decomposition is nothing new in software design, On the Criteria To Be Used in Decomposing Systems into Modules, D.L. Parnas, from 1972 is discussing the same challenges. They reach the conclusion “(…) it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart. We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others.“

With what-changes-together-lives-together in the back of the head, the parts of the system we have moved to the new API is recommendations, playback (avspilling), radio-specific views and a catalog of programs/series. I would argue many strategies leads to similar results. In Domain Driven Design this would typically be a set of bounded contexts. Uncle Bob calls this Screaming Architecture. Jimmy Bogard has a presentation onArchitecture in slices, not layers. I would argue these are all quite similar concepts, and that this is a good sign. If many approaches leads to similar partitioning strategies, this strengthen my trust in the chosen strategy.

There are still concepts in the codebase that are not domain-concepts. No architecture is perfect, and our is far from it. Still, if you want to do changes to recommendations, you probably have an idea of where to start looking.

Domain concepts can be recognized in the codebase

Assembly and namespace dependencies new codebase

At an assembly level everything seems to look ok, just as with the technological strategy. It is seldom here, at the highest level, you will see the results of the wrong abstraction. The assemblies marked with red boxes are separate, functional units (Playback, recommendations, the program catalog etc). There are definitely still challenges with naming and structuring of the code base, but at least domain concepts can be recognized at our first level of modularization.

Assemblies in new solution

 

By showing all namespaces and their dependencies, the real complexity of the solutions materializes. By building the new solution with defined NDepend’s rules disallowing mutually dependent namespaces from the beginning, we have been able to make a directed graph. To be fair, there is less functionality in this new API than in the previous one, but this still gives some indication that complexity is lower.

 

Namespaces in new solution

Focusing on a domain concept in the new solution

Now, if we do the same exercise as before, trying to find the namespaces used by the playback (avspilling in Norwegian) module, the picture is much more clear. This is actually a graph we could print and have discussions around. There are no shared “Controller”, “Manager”, “Services”, “Interfaces”, “Repository” modules that tie playback to the other parts of the system!

Namespaces in playback

Techincal debt in new solution

Again the standard NDepend rules for technical debt indicates that we are at a much better place with the new architecture.

Tech debt in new solution

Summing up

To me, the answers are clear. I will never again use technical layers as a primary strategy of decomposition (technical layers inside functional boundaries are still ok). Continuously refactoring to separate functional modules is still hard work, but the horror image of our big ball of mud should help us stay strong and steer in the right direction.

8 kommentarer

    • Bjørn Einar Bjartnes (NRK) (svar til Erik Assum)

      Absolutely! Or if the new radio team get the need to deploy more independently, making a new service (Microservice…?) and moving it out into a separate deployable is pretty simple too.

  1. This question might be slightly off-topic, but since you mentioned APIs for NRK Radio:

    I’ve been looking for a way to extract the playlists from various radio shows (e.g. what you can find here: radio.nrk.no/serie/nitimen/MUHR01010617/04-05-2017). I asked NRK’s Publikumsservice if there exists any such public API, but they weren’t able to provide me with an answer.

    So, mostly for fun, I started writing a parser to manually search and extract playlist information from radio.nrk.no (I’m thinking further integration with Spotify/Tidal to generate playlists etc.). But if such a thing already exists, that would save me some work…

    Svar på denne kommentaren

    • Bjørn Einar Bjartnes (NRK) (svar til Ole K)

      Short answer:
      Yes, there is an API.
      No, it is not public.

      I will make a wild guess your follow up answer is «can you make it public?» Codewiseit is really use to do… My answer then would have to be, I need to ask some people… And I also have a follow up question, what are you going to use it for?

Legg igjen en kommentar til Ole K Avbryt svar

Din e-postadresse vil ikke bli publisert. Obligatoriske felt er merket med *. Les vår personvernserklæring for informasjon om hvilke data vi lagrer om deg som kommenterer.