I work with JSON a lot. For instance, I write tests for the TV Streaming API that serves JSON to our clients. In those tests, I typically want to verify that the API returns a JSON response that matches my expectations.
For instance, say I expect the JSON response for given a TV show to look like this:
How can I verify that the response matches my expectation? The answer is obvious: I can write a test to check. It should be easy, right? I just call the API to get the actual response, and compare it with the expected response.
When I call the API, I get the following response:
Are they equal? Well, they look quite similar, but it’s hard to tell, right? I better check to make sure. But exactly how should I do that? How should I carry out the comparison?
Should I compare the responses as strings, or should I deserialize them to some other data structures and compare those? Both options operate at wrong levels of abstraction. Either I make a very low-level string comparison that ignores all the structure of JSON documents, or I make a proxy comparison of something I derive from the JSON documents.
What I really want to do is to make a comparison of the JSON documents themselves. This should be easy to do, since JSON is small and well-defined. But it means I need a tool that understands JSON. Such a tool should be able to point out exactly the differences (if any) between my expectations and the actual JSON response. Yet I haven’t been able to find any such tool that I can use! Our API runs on .NET in Azure, and in the .NET space, it seems there are none.
To fix this problem I wrote a little JSON diff library called Quibble. It takes two text strings, parses them into an internal JSON structure, compares the structures recursively property by property, and returns a list of any differences that it found along the way. It uses JSONPath syntax to point out exactly where in the JSON strings the differences can be found. Basically Quibble gives you a list of the needles it found and their locations in the haystack.
Quibble distinguishes between four kinds of differences:
- Type: e.g. string vs number.
- Value: same type but different value, e.g. the string video vs the string audio.
- Object: when two JSON objects have differences in their properties.
- Array: when two JSON arrays have different items.
By calling JsonStrings.Diff you get a list of Diff values that you can iterate over and pattern match against. If all you want is a textual description of the differences, you can call JsonStrings.TextDiff instead.
Now that I’ve made the tool I was missing, how can I use it to write better tests? Good question.
What I really want to do is assert that the actual JSON response matches my expectations. If it doesn’t, I want the test to fail with a precise explanation that allows me to figure out what is wrong. To this end, I wrote another little library called Quibble.Xunit. It provides a method JsonAssert.Equal that uses Quibble to compare two strings for equality. If the strings do not contain the same JSON content, Quibble.Xunit will throw a JsonAssertException and point you to the differences.
Here’s a simple Xunit test for my use case:
The test should be easy enough to read. What happens when I run it?
Alas, the test fails! Turns out there were a number of tiny issues with the response from our API.
The output from the test run looks like this:
Note that Quibble.Xunit points out the number of differences, where to find them, and what they are. It makes locating and fixing any problems so much easier.
I think Quibble and Quibble.Xunit will be very useful for us. If you find yourself in a similar situation, perhaps they can be useful for you too.