Chapter 2 of 12
What counts as knowing?
Written edition. The audiobook has an earlier version of this page.
On this page
Before judging a person from a pattern, separate four questions: what happened, what might happen next, why it happened and what you should do. Evidence that answers one may not answer the others. This chapter helps you identify the missing information without postponing every decision.
One habit to practise
When a pattern makes you want to change a rule, ask: “What else could explain this pattern?”
Description, prediction and explanation are different
A description reports an event. A prediction uses a pattern to anticipate another event. An explanation proposes why the event happened. A decision also involves costs, responsibilities and what people consider fair.
Brief fictional example: People who reply quickly to an invitation attend more often. Reply speed may predict attendance without causing it: predictable work hours could make both easier. Requiring fast replies might exclude people rather than increase commitment.
Julia Rohrer’s 2018 methods paper explains how researchers check claims about cause and effect when they record what happens without assigning people to different groups. A link between two things can arise because one affects the other in either direction, because a third factor affects both, or because people with different habits enter the groups being compared. Calculations that account for other measured factors still depend on assumptions about how those factors are related. Read the paper.
Check what the record includes
Ask what was measured, whose experience is represented and what an empty entry means. A missing purchase is not necessarily a rejected offer. An attendance list may omit people who were never invited. Several people repeating one account do not supply several independent observations.
Check one detail that could change your decision. This is often more useful than collecting more examples from the same incomplete source.
Count false alarms as well as detections
A warning is easier to interpret when you know how common the event is and how often the warning is wrong.
Here is an invented arithmetic example, not a real screening tool. Of 1,000 people, 10 deliberately broke a rule. A warning flags 8 of those 10 and wrongly flags 99 of the other 990.
Definition
- Base rate
-
How common something is in the relevant group before you consider a particular clue or test result.
Example: In Chapter 2’s invented group, 10 of 1,000 people deliberately broke the rule. The base rate is 1%.
| In this invented group | Flagged | Not flagged |
|---|---|---|
| Broke the rule: 10 people | 8 | 2 |
| Did not break it: 990 people | 99 | 891 |
Look down the Flagged column. There are 107 flagged people in all, and 8 of them broke the rule (about 7.5%). Now look across the Broke the rule row. The warning caught 8 of the 10 people who broke the rule (80%). The two percentages answer different questions: how often a warning is correct, and how often the warning catches a rule-breaker.
In 1995 experiments, Gerd Gigerenzer and Ulrich Hoffrage found that presenting probability information as counts could improve reasoning on the tasks studied. The example uses that presentation; its numbers say nothing about dishonesty in your circumstances. Read the paper.
If the starting frequency is unknown, leave it unknown. Do not invent a precise probability about a person. A low population frequency also does not erase specific evidence of harm.
Update confidence without inventing certainty
New information can weaken one explanation without proving another. “I have less reason to think this was deliberate” is a legitimate update. So is “The record supports the event, but intent remains uncertain.” Decide in advance what would count against your preferred account.
Match the checking to the decision
Ask whether another check could change what you do. A reversible booking change needs less certainty than a public accusation. Facts can reveal consequences without deciding which trade-off is fair. Where harm is plausible, support or protection need not wait for certainty about motive; see Chapter 11.
Go deeper
Observation and interpretation · Comparing explanations · Uncertainty.
Key takeaways
- Description, prediction, cause and decision are different questions.
- Check missing cases, source independence and false alarms.
- Update your explanation and use proportionate checks; uncertainty need not prevent action.
Put it into practice
Recall
Without looking back, why does a pattern between two things not show that changing one will change the other?
Show the useful distinction
One thing could affect the other in the opposite direction. Both could have another cause, or the groups could have differed before the comparison. Ask: “What else could explain this pattern?”
Try it
A new fictional situation: A café finds that loyalty-card holders spend more. The owner wants to give everyone a card to increase spending.
What do we know about spending now? Which customers’ earlier spending would you compare to check whether the people who chose a card already spent more? Say or write your answer before opening the example.
Show an example answer
Current cardholders spend more than customers without cards. Compare the two groups’ spending from before the current cardholders got their cards. If the future cardholders already spent more, the difference existed before they joined. That comparison leaves open whether getting a card also changes spending.
Use it this week
This week, examine one ordinary claim that two things are causally connected. Ask what else could explain the pattern and check one decision-relevant fact. Record what changed in your view and what remains unknown. Do not impose an experiment on someone else.
My practice notes: Check a pattern before explaining it
Notes save automatically in this browser when storage is available. Avoid names and sensitive details, especially on a shared browser. Download a text copy before clearing your browser data.
More practice (optional)
Another comparison
A separate invented example: A warning flags 20 bookings; only 2 are confirmed problems. What can you say about flagged bookings, and what can you not say about all bookings?
Show what to consider
Two of the 20 flagged bookings are confirmed problems. You still need information about unflagged bookings and the completeness of confirmation before assessing missed cases or the overall problem rate.
Go deeper in the reference library
- Compare occasions & test an explanation — Investigate recurrence, exceptions and competing explanations through proportionate observation, action and review.
- Use AI without reinforcing a first impression — Use an LLM to organise questions and alternatives while keeping generated suggestions separate from observations.
- Changing criteria & moving goalposts — Distinguish justified revisions from repeated changes that make an agreement impossible to satisfy.