Direkt zum Hauptbereich

Noise: reducing judgment flaws in your organization

Just as the number of citations to Thinking, fast and slow/1/ were starting to ebb, Daniel Kahneman has just published a new, big book that will surely be widely read and quoted. Written together with HEC Professor Olivier Sibony and Harvard Professor Cass Sunstein, Kahneman's Noise: A flaw in human judgment/2/ is a highly readable work on a little-recognized human problem, namely that professional judgments and decisions contain unacceptable levels of variance, even though we expect them to be fair, reasoned and deliberative. The amount of noise in our world has profound implications for society and organizations. Fortunately, Kahneman et al. offer workable remedies. Their conclusions indicate once more how structured cooperation in teams can offset our human fallibility. 

[Deutsche Version]

What is noise?

Figure 1: bias vs. noise 
Whereas the psychological biases charted in Kahneman’s previous work are individual, error-prone heuristics that shape thinking, noise by their definition is “the unwanted variability of judgments”. It is systematic variance both within an individual and within organizations that amount to errors (under the assumption that the judgment is aiming for some kind of truth). To make the difference between bias and noise clear, they present Bernoulli’s classic target diagram as seen in figure 1. Whereas bias systematically misses the target in the same direction; noise scatters widely around the bullseye. Of course, a judgment problem can be prone to both simultaneously. 

The authors’ concern is human judgment, in other words any situation where a person is evaluating, assessing, predicting or deciding: “judgment is a measurement where the instrument is the human mind”. We rely on judgments extensively. The authors cite numerous examples: judges in courts, managers evaluating investments, forensic analysts comparing fingerprints, underwriters calculating insurance rates, HR departments assessing job candidates, teachers grading students etc. As these examples show, judgments are important because they are social; they occur in social and political institutions and in organizations of all kinds; and they have real consequences for those involved. (Note: they use the word “judge” to mean any person making a judgment, not just judges in a courtroom.)

Why noise matters

The authors contend that noise is pervasive; it can have high magnitudes; it can be costly financially and in human terms (reputation loss etc.); and it can undermine our trust in the institutions charged with judging. They cite egregious examples where people convicted of similar crimes receive vastly different sentences. Being good scholars, they don’t just mention examples, they cite study after study, even metastudies with hundreds of thousands of court cases showing wide variance in the verdicts. In financial contexts, such as insurance underwriting or investment decisions, the authors reveal huge ranges for costs and pricing. To those who argue, “but if the price is sometimes too low and sometimes too high, it averages out”, they point out that the error does not average, rather it aggregates: a low price leaves money on the table, and a high price cause customers to go elsewhere. 

The book campaigns to reduce noise. Some methods for doing so already exist. Rule-setting, for example, is a simple way of structuring judgments that drastically reduces noise by imposing narrow decision boundaries. Other techniques, for example regression analysis or AI algorithms reduce noise and consistently beat human judges, especially at predictive judgments, if measured against whether the prediction came true (for example, a new hire staying with the company for several years). This has been known since the 1950s and confirmed in hundreds of studies over a broad range of topics./3/ Nevertheless, we generally reject the rigidity of rules or leaving judgment to machines, even in cases where the machine produces manifestly better predictions or more consistent and fair results./4/

Human beings are simply flawed: we are prone to the biases catalogued by Kahneman and others. We fall prey to the illusion of validity, which means that we conflate the available evidence into our ability to predict, for example, future performance. As we cannot know the future, there are fundamental uncertainties that we tend to ignore. Moreover, we are highly adept at constructing coherent explanations even in the face of conflicting or missing information. This construction generates an internal psychological reward that feels good. Cognitively, the feeling becomes confounded with the belief of knowing, thus contributing to the “gut feeling” that executives frequently cite in justifying their decisions. Finally, we understand our world primarily through causal thinking and by linking events into stories. In hindsight, we build coherent narratives that appear as logical chains of events, thus creating the illusion that we had anticipated the result, even though we had no way of seeing into this future.

One trouble with noise, the authors note, is that it is a phenomenon that can only be seen clearly through statistical evaluation. We might chafe when a teacher seems to grade a student’s work unfairly. We’ll condemn the grading as biased. Noise, however, is the systematic error that arises through a range of teachers who grade according to different standards or personal values. We can only see the extent of the noise statistically, even though its existence affects each student. (A fine technical point: the psychological biases of individuals is a factor in the total noise of an organization.) Thus, mechanical tools to reduce noise act likewise at the statistical level. Even though they are often better, we resist them as inhumane, too rigid, or offensive to our self-worth as experts.

Reducing noise in organizations

Despite the importance and popularity of Kahneman’s work on biases, it is unclear whether we can do much to avoid the pitfalls of overconfidence, the availability bias, the planning fallacy, or anchoring effects. Kahneman himself admits, “my own experience of how little this knowledge [on human bias and fallibility] has changed the quality of my own judgment can be sobering.”/5/ That sentence made me feel better about my own struggle against some of the biases on Kahneman’s list. Nor do individuals have much of a chance against noise. Kahneman continues, “Avoiding noise in judgment is not really something individuals are going to be very good at. I really put my faith, if there is any faith to be placed, in organisations.”/5/

In the end, Kahneman, Sibony and Sunstein remain steadfast in their appeal to reducing noise. Since human beings remain skeptical about turning judgments over to machines, the authors see it as ever more imperative that we reduce noise in human judgments. They recommend solutions that function in organizational contexts. From the perspective of our mission here at Teamworkblog, this is where the book becomes especially interesting. It adds substance to the platitude: “together, we become stronger,” or rather, smarter.  (Alas, one conclusion from the book is that until now we haven’t really done so.)

The authors’ recommendations center around structuring our decision-making processes through the use of good “decision hygiene”. They propose sticking to the following six principles:

  1. Judge accurately. While values and creativity play a role in the setting of goals to which a decision contributes, they introduce noise when they affect a judgment.
  2. Think statistically, using tools such as base rates, reference classes or the statistics of similar cases etc. These items can frame the judgment case, without reducing it to mechanical rules.
  3. Structure judgments into separate tasks. They advocate a tool called a "mediating assessments protocol" that seeks fact-based assessments of components before consolidating them into a final decision. Combined with Principle 5 and supported by Principle 2, such a structured process can lead an organization to a much less noisy decision. 
  4. Avoid premature judgments that are no more than intuitions. This principle includes sequencing information so that the expert is not biased by other information.
  5. Use multiple opinions where possible, and aggregate them.  Ensure that the individual judgments are independent and remain unknown to the others. Among other things, they recommend an "estimate-talk-estimate" procedure that closely resembles Planning Poker known to agile teams.
  6. Favor relative judgment and relative scales. Human beings are simply much better at pairwise comparison than at abstract categories and scales.

Concretely, I don’t see unsurmountable challenges for an organization that wants to reduce its noise to finding the processes and structures needed to implement these principles. We need structures in order to avoid repeating our mistakes. They allow us to be smart and creative—together. Structure need not be bureaucratic. Without structure, we have chaos and noise. We need more statistical bases and awareness in contextualizing decisions. These steps are not hard. Where there’s a will, there’s a way.

There is much more to this book than I have been able to discuss here. Noise is an important concept and the book is a great exposition of the topic. 


/1/ Daniel Kahneman, Thinking, Fast and Slow (Farrar Straus & Giroux, 2011).

/2/ Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgment (New York: Little, Brown and Company, 2021).

/3/ One main reason is simply that mechanical systems remove noise. They may still be biased, but they are not noisy. The amount of error and variability introduced by human judgment is so high, that simply removing the noise gives better results.  We think that human beings are better at managing the subtleties and complexities of a situation than any mechanical model could. But, the research shows often high correlations between the various variables, such that the addition of more does little to improve the decision making. Thus, a noiseless regression model that relies on just one or two key variables will achieve a better result, even when no weighting is applied.

/4/ The authors acknowledge these objections and treat the issue of objections to their argument sensitively. They appear themselves ambivalent about a technocratic world of AI. They note, too, that deep learning algorithms can still be racially biased, even when a it was programmed specifically to avoid any kind of racial profiling (in the USA, for example, postal code correlates closely with race).

/5/ Tim Adams, “Daniel Kahneman: ‘Clearly AI Is Going to Win. How People Are Going to Adjust Is a Fascinating Problem,’” The Guardian, May 16, 2021.  Kahneman is a modest man of science, ever egolessly pursuing the truth. By all accounts, he delights in discovering that he was wrong. As he once stated to Adam Grant, “Being wrong is the only way I feel sure I've learned anything.” Adam M. Grant, Think Again: The Power of Knowing What You Don’t Know (New York, New York: Viking, 2021), p. 61-62.


Beliebte Posts aus diesem Blog

Das Ubongo Flow Game

Spiele bieten eine gute Gelegenheit, zeitliche Erfahrungen zu verdichten und gemeinsam zu lernen. Karl Scotland und Sallyann Freudenberg haben im Mai 2014 das Lego Flow Game veröffentlicht. Wir haben die Spielidee übernommen, aber das Spielmaterial gewechselt. Statt Legosteinen benutzen wir Material aus Grzegorz Rejchtmans Ubongo-Spiel. Hier präsentieren wir die Anleitung für das Ubongo Flow Game.

Outlook-Aufgabenliste: bitte nicht die Aufgaben des ganzen Teams!

Am Tag der Arbeit kommt eine Lösung, nach der ich schon so oft gefragt wurde: Wie schaffe ich es, dass meine Outlook-Aufgabenliste nur meine eigenen Aufgaben anzeigt und nicht auch die E-Mails, die meine Kollegen gekennzeichnet haben oder Aufgaben, die einfach in einem gemeinsamen Postfach stehen?

Kategorien in Outlook - für das Team nutzen

Kennen Sie die Kategorien in Outlook? Nutzen Sie diese? Wenn ja wofür? Wenn ich diese Fragen im Seminar stelle, sehe ich oft hochgezogene Augenbrauen. Kaum jemand weiß, was man eigentlich mit diesen Kategorien machen kann und wofür sie nützlich sind. Dieser Blogartikel stellt sie Ihnen vor.

Und jetzt alle zusammen! Teams - OneNote - Aufgaben - To Do

Ein Meeting jagt das nächste. Sich da nicht zu verzetteln, wird  im Zeitalter virtueller Besprechungen  noch anspruchsvoller. Kein Wunder, dass  im Zusammenhang mit Microsoft 365  zwei Fragen besonders häufig auftauchen: Wie dokumentiert man Besprechungen gut? Was hilft, offene Aufgaben nachzuhalten? Eine gute Lösung: Das in MS Teams integrierte OneNote-Notizbuch als gemeinsame Plattform auch für den Aufgabenüberblick zu nutzen.

Das neue Outlook - One Outlook - erster Eindruck

Microsoft hat ein Problem: Outlook ist nicht gleich Outlook. Da ist das gute alte Outlook in der Desktop-Version. Das ist das, womit fast alle von uns im Alltag arbeiten und worüber ich hier schon oft berichtet habe. Outlook auf dem MAC sieht aber anders aus. Outlook auf Mobilgeräten sowieso. Dann gibt's noch Outlook im Web. Kein Wunder, dass Microsoft das alles entwirren, verschlanken und vereinheitlichen möchte. Gelingt es? Hier die interessantesten Funktionen des neuen Outlooks . 

Beispiel für eine Partyplanung mit Scrum

Wer sich neu mit Scrum beschäftigt, ist vielleicht überwältigt von den ganzen Fachbegriffen. Dann sieht man vielleicht gar nicht, wie einfach die einzelnen Elemente von Scrum sind. Deshalb hier ein einfaches Beispiel für die Vorbereitung einer Party mit Hilfe von Scrum.

E-Mail-Vorlagen gemeinsam nutzen (Outlook)

Mittlerweile wird praktisch alle Routine-Korrespondenz in Outlook erledigt. Was liegt da näher, als ein gutes Set von Vorlagen zu erstellen und diese gemeinsam in Team zu nutzen? Leider hat Microsoft vor diesen – an sich simplen – Wunsch einige Hürden gebaut.

2 Tätigkeiten, bevor man agile Arbeitsweisen nutzt

Wenn Teams und Organisationen sich mit Agilität beschäftigen, hoffen sie auf eine schnelle Lösung für ihre Probleme. Mit Scrum und Kanban sollte es doch sofort und spürbar besser werden. Abgesehen davon, dass diese Teams vielleicht noch nicht wissen, warum Scrum und Kanban funktionieren, sind aus meiner Sicht zwei wichtige Fragen noch nicht beantwortet.

Protokolle in OneNote - neue Ideen für's neue Jahr

Protokolliert Ihr Team seine Besprechungen in OneNote? Das geht einfach, schnell ist teamfähig und hat eine exzellente Suchfunktion. Die beliebte Fragen "Wann haben wir eigentlich beschlossen, dass..." ist so schnell beantwortet. Darum wird OneNote an dieser Stelle immer beliebter. In meinen Seminaren dazu sind gute Ideen entstanden, die ich hier weitergeben will.

Führung weiterentwickeln mit Surflehrer Pepe (Pedro)

Führung weiterentwickeln mit Surflehrer Pepe  "Du bist für dein Board verantwortlich, ich bin für dich verantwortlich. Wenn du genau das tust, was ich dir sage, werden wir und alle anderen Leute hier sicher sein und wir werden eine wunderbare Zeit haben." - Pepe (Pedro) Surflehrer in Ericeira Gute Führung hängt stark vom Kontext und der aktuellen Kultur des Unternehmens ab. Der Stil von Pepe unterschied sich von den anderen Surflehrern, die ich zuvor hatte. In diesem Beitrag erzähle ich euch mehr von meinem Surfkurs, wobei ich mir nicht sicher bin, ob es eine Geschichte über meinen Surflehrer oder über Führung an sich ist.  Let’s grow, Michael