nrk.no

With a quiz to comment, readers test their article comprehension

Kategori: English-articles


Since February our readers must pass a mandatory quiz to comment on our articles. But the world famous quiz has found another use too.

First, some navel–gazing and context.

NRKbeta is the Norwegian Broadcasting Corporation’s (NRK) tech-site and R&D-blog. This February we launched a new experiment.

Readers wanting to comment on a story had to pass a mandatory test about the articles content.

Despite other websites abandoning their comments section, ours has mainly remained a positive space the last ten years.

Comments from our audiences have solved difficult problems for us. It has gathered fantastic and crazy ideas from our audience that we have tried out. It’s even helped us hire talented people! In short: a valuable arena we dedicate a lot of time and effort to maintain and improve.

But as some of our most read stories have ended up with discussions below average, we suspect some readers just skimmed the headline before heading for the comments section. That is not what it is for.

That is how the idea of a speed bump in the form of a quiz originated.

We hoped this would contribute to eager commentators cooling down. It could give them the opportunity to think things through before punching away at the keyboard.

Around the world

A short while after we went live, Harvard’s Nieman Lab picked up on it and wrote about it. Joseph Lichterman’s story became one of their most read and started a tsunami of international attention.

The experiment recieved praise from prominent personalities in the media industry world wide. Soon CNN, France24, BBC and Gizmodo picked up the story as well. Then we received a ton of interview requests from radio, TV, newspapers and news sites from all over the world.

Our small newsroom took turns answering questions about how horrible our comments section must have had been (no, it really was not), if the quality has improved (almost impossible to say) or if the experiment had been a success (we do not know, it has just been live a few days).

We kept note to about 300 articles were published about it, and can still feel the shockwaves.

Just a few days ago, a link to the Indepentent’s story from early March received over 86 000 upvotes on reddit. A tweet from Wall Street Journal’s tech columnist received over 5 000 retweets and close to 8 0000 likes. Not bad on Twitter.

TIL if you want to comment on the Norwegian news site, NRKbeta you must first take a quiz to test your basic understanding of the article. This is done to prevent ranting and foster positive conversations from todayilearned

The feedback from colleagues in the media industry and most of our readers have – as mentioned – been very positive. But this is just the initial reaction to the concept and us trying it out. Not really to to the results. Results we did not have when the questions started pouring in.

The combination of a media industry desperation for new innovation, poor comments sections and an apparent easy and smart solution proved to constitute a potent viral mix.

So let’s try to summarize what we have learned so far, almost six months after we went live.

In our own comments section the quiz has of course been diligently discussed. The majority has been positive, but at least one of our readers thinks the quiz is a bad idea. Details like if a conference is called F8 or Eight has nothing to do with «if you are qualified to comment what the article actually is is about«, the reader writes.

Another suggests an improvement and adds «Kind regards one writing in a comments section for the first time in ten years with new optimism!«.

We have taken a closer look at the numbers, but there isn’t much logic or clear answers to report. At least not if assume that everyone taking the quiz actually wanted to leave a comment.

On average, there is a lot more attempts – both correct and wrong – than actual comments.

It seems many take the quiz to check how much they remember from the story – and not necessarily to leave a comment. Almost as a fun little game after reading.

Comments, correct and wrong answers to 14 quizzes Illustration: NRKbeta

A story that stand out is our explainer on how to like Facebook statuses with a rainbow in connection to pride. There were over a thousand wrong attempts to answer the quiz. Due to a human error, the right answer to one of the questions was not indicated. It made it impossible to pass the quiz. Hence the many logged wrong attempts.

On average, there is a staggering error rate of 72% on the quiz. We also suspect a lot of wrong answers coming from visitors of faraway lands. Most would have a hard time breaking our encryption made of solid Norwegian language.

The recent wave of visitors from the Independent’s story via Reddit, were met with our most recent article. We did not expect it would have traffic above normal, and did not see the need to enable the quiz.

Several left comments asking about the quiz. Thus we assume most international visitors have not understood the quiz. Likely they wanted to play with the functionality (and lucky for you we open sourced it).

We have not published a large amount of stories with a quiz, making the sample small. It’s one of the reasons why we have not established any set rules for when the quiz should be enabled. The idea was to test it on stories that had potential for a gloomy comments section. It is something we are proud to rarely have here at NRKbeta.

Colleague Anders highlights the fact that a quiz raises the bar to leave small comments. Popping in to say «fun piece» or «hey spitfire, chill out» before leaving the article is now a bit harder.

This favors the most eager with the most time on their hands. From time to time this has lead to a decline in quality and tone, causing him to often abandon the quiz module. If you have tried the quiz module, we would like to hear your experience. Leave a comment below 😉

In other words, it is hard for us to present evidence concluding with success or failure. But the numbers seem to show that the quiz has worked like a little game for many readers. They like to take the quiz, but not to leave a comment.

Being tested on how much they remember from the article seem to be the most popular use of our quiz.

25 kommentarer

  1. I think this is a brilliant idea. The problem of how to prevent echo chambers while also preventing trolls and flamers has been a vexed one. This eliminates bots (assuming you can stop someone feeding the correct answers into a front end) and certainly stops ideologues with knee-jerk responses. I would imagine, however, that it will take additional time to get a story published, and to get the maximum benefit you would need to include questions that test understanding (say of a speech or policy) as well as simple «what colour tie» type factual questions.

    Well done, and thank you!

    Svar på denne kommentaren

  2. Early Results from NRKbeta’s Comment Quiz Plugin Show Readers Enjoy the Quiz but Rarely Leave a Comment – WordPress Initiate

    […] has published some preliminary results after six months of experimenting with a mandatory quiz before […]

    Svar på denne kommentaren

  3. Early Results from NRKbeta’s Comment Quiz Plugin Show Readers Enjoy the Quiz but Rarely Leave a Comment – AmDee WordPress Headlines Site

    […] has published some preliminary results after six months of experimenting with a mandatory quiz before […]

    Svar på denne kommentaren

  4. Commentard Quizwall experiment ends with more quizzing than commenting | Crawfordwise

    […] quiz-to-comment was only introduced on a subset of stories, but now NRK has published some findings from the […]

    Svar på denne kommentaren

  5. Il sito dove si può commentare un articolo solo se l’hai capito: com’è andato l’esperimento – Notizie Oggi

    […] che l’idea dell’emittente norvegese NRK stata lanciata nel marzo 2017, dopo quasi sei mesi possibile guardare ai primi risultati […]

    Svar på denne kommentaren

  6. This really is a brilliant idea. At least it is thinking outside the box when trying to combat ranting, and people commenting after only reading the headlines.

    It was mentioned on the latest episode of Adam Ruins Everything as well, though not by name, which is why I am here to have peek 😀

    Episode is season 2 episode 8, in case you are curious.

    Svar på denne kommentaren

  7. Wonderful idea. I think it is actually very tweakable, you can play with parameters for a desired effect. For example you can allow the comment in a «purgatory» and post it only if the quiz is passed, so you’ll have more comments if you like. The quiz may be created by the author, having the main ideas he or she put in the article checked. It may become a new way of writing articles in online, the article in itself and the «lock» made by the author. Or the ones who comment constantly could be ranked levels of access, like a sort of editors, so they could be allowed to create that filter-quiz. Wonderfully simple idea with so many potential implications. It should be implemented as cross-platform, FB integrated, twitter, etc..Congratulations!

    Svar på denne kommentaren

  8. ჯერ ქვიზი – შემდეგ შეგიძლია კომენტარიც დაწერო – JRC

    […] შესახებ წერს. 2017 წელს გამოცემამ ექსპერიმენტი […]

    Svar på denne kommentaren

Vil du kommentere? Svar på en quiz fra saken!

Vi er opptatt av kvaliteten på kommentarfeltet vårt. Derfor ønsker vi å sikre oss at alle som kommenterer, faktisk har lest saken. Svar på spørsmålene nedenfor for å låse opp kommentarfeltet.

Who wrote about the quiz first?

Is the quiz module open source?

Where did a link to our story recently get 86 000 upvotes?

Legg igjen en kommentar

Din e-postadresse vil ikke bli publisert. Obligatoriske felt er merket med *. Les vår personvernserklæring for informasjon om hvilke data vi lagrer om deg som kommenterer.