Better questions,
fewer false certainties.

Illustration Two colleagues argue for forty minutes about whether “meetings are a waste of time.” One is thinking of the Monday status call, where eleven people listen to updates they could have read. The other is thinking of the rare meeting where a tangled decision finally got made. One measures waste in minutes; the other in decisions that never happen without a room. Neither is lying, and neither will win, because they are not disagreeing about an answer. They are holding different questions.

This happens constantly — in staff rooms, in family arguments, in product launches, in comment sections. We reach for evidence, statistics and anecdotes before noticing that the sentence we are fighting over could mean three different things, that half of it is a prediction with no date, and that the other half is a moral judgement wearing the costume of a fact.

Interpretation Questioncraft is the habit of repairing the question first. It asks: what exactly is being claimed, about whom, compared with what, and what would count against it? It is not a trick for winning. Done honestly, it frequently reveals that you agree more than you thought — or that you disagree about something deeper and more interesting than the thing you were shouting about.

There is a trap on the other side, though. A question can be “clarified” until it is so narrow that it no longer matters, or forced into a number when the important part was never numerical. So this studio has two jobs: make claims testable where testing helps, and protect the ambiguity that is doing real philosophical work.

Most attractive claims are vague assertions: they sound like they say something but bundle several questions together. Pulling them apart is the single most useful move in this studio. Take a sentence you have probably heard: Remote work is killing company culture.

Arguments loop when people answer different rows. The manager citing departure rates is answering the predictive row; the employee talking about getting her evenings back is answering the normative one. Both are right about their row and talking past each other.

Established framework The gap between the last row and the others is old: philosophers have long distinguished claims about how things are from claims about how they ought to be — a distinction usually traced to David Hume. Facts matter enormously to value questions, but they rarely settle them alone. The five-row table itself is this studio’s teaching device, not a formal standard; real questions often mix shapes, and that is fine once you can see the mix.

Illustration “Students today can’t concentrate.” Before arguing, ask what you would have to see to agree. Concentrate on what — a lecture, a novel, a video game? For how long? Compared with which students, measured how, at the time? If the comparison is with our memory of ourselves at fifteen, we are comparing a sample of one, filtered by nostalgia, against everyone else’s children.

Specifying the observations that would bear on a concept is called operationalisation. Established framework The physicist P. W. Bridgman pushed the idea hard in 1927, suggesting a concept means no more than the operations used to measure it. The philosophical history is instructive: critics argued that operations never exhaust what a concept means, and that many good scientific concepts have no single operational definition. Psychologists later settled on a more modest use — operational definitions as partial, provisional indicators rather than the whole meaning.[Chang]

That modest version is the one worth keeping. Operationalise to make a claim checkable, then remember that the measure is a stand-in. A school that tracks “concentration” as minutes of silent reading logged per week may soon have a great deal of logging and very little reading.

Established framework The social scientist Donald T. Campbell warned about exactly this: the more a quantitative indicator is used for important decisions, the more it tends to be distorted, and the more it can corrupt the very process it was meant to monitor.[Campbell] So the better question is rarely “what number proves this?” It is “which few observations, taken together, would make me more or less confident — and how might people game each one?”

Illustration Suppose a report finds that schools with more tablets per pupil post better exam results, and a headline concludes that tablets raise results. Before accepting or rejecting it, list what else could produce the same pattern:

• A confounder. Better-funded schools can afford tablets and smaller classes, tutors and families with more time at home.
• Reverse causation. Schools with strong results may win grants that come with devices.
• Selection. Perhaps only schools pleased with their tablets returned the survey.
• Chance. With a handful of schools, a pattern this size can appear from noise.

Established framework Causal questions are always comparative. Asking whether tablets “work” really asks: what would these pupils’ results have been, in these schools, without them? Epidemiologists Miguel Hernán and James Robins argue that a causal question should be stated precisely enough that you could describe the hypothetical experiment that would answer it — and that the roles of data and of assumptions should be kept visibly separate.[Hernán & Robins]

A useful everyday version: before arguing about a cause, write one sentence describing the fair comparison you would want. “Similar schools, similar intake, half given tablets, results compared after two years.” You may never run that study. But writing it makes the rival explanations obvious, and it shows you how far the evidence in hand falls short of it.

Illustration Picture a neighbourhood shop that rearranges its shelves. A month later the owner says, “The regulars love it — nobody has complained.” The sentence is sincere and nearly empty. The people who disliked the layout and quietly started buying elsewhere are not there to complain. The evidence has been filtered by the outcome it is supposed to test. (This is a hypothetical, of the kind familiar to anyone who has spent time behind a family counter; it reports no actual shop’s results.)

Every body of evidence has a doorway, and the doorway decides who gets in. Survey respondents are people who answer surveys. Online reviews come disproportionately from the delighted and the furious. Success stories come from survivors. “Everyone I know” is a sample chosen by friendship.

So a refined question names its population and its doorway: Of customers who bought from us at least monthly before the change, how many bought at least monthly in the two months after? That question can be answered from a till roll, and it includes the people who left. It still cannot say why they left; that needs another question, and probably a conversation.

Established framework Karl Popper argued that what makes a theory scientific is that it could, in principle, be shown false: it forbids certain observations. Confirmations count for much only when they come from risky predictions that might have failed. A claim compatible with every possible outcome tells you very little about the world.[Popper / SEP]

Applied to everyday claims, this becomes a simple and slightly uncomfortable question: what observation would make you abandon or weaken this? If someone says a colleague is “secretly hostile” and reads both friendliness and coolness as proof, the claim has stopped saying anything. It has become a lens that colours every observation.

Falsifiability is not the whole story, and the limits matter. Philosophers such as Imre Lakatos and Hilary Putnam pointed out that a failed prediction rarely kills a serious theory on its own, because predictions depend on auxiliary assumptions — about instruments, conditions, measurement — any of which might be the thing that failed.[Popper / SEP] Real inquiry weighs patterns of evidence over time rather than executing a theory after one miss.

Evidence The danger also runs the other way: flexible analysis can manufacture support. Joseph Simmons, Leif Nelson and Uri Simonsohn showed with simulations and two experiments that undisclosed choices — which measures to report, when to stop collecting data, which covariates to include — can sharply inflate the chance of finding a “significant” effect that does not exist.[Simmons et al.] John Ioannidis’s widely discussed 2005 essay modelled how small studies, small effects, flexible designs and competing interests reduce the chance that a published positive finding is true.[Ioannidis] The everyday lesson is modest: decide what counts before you look, and say how many ways you looked.

Not every good question is testable, and not every testable question is good. “Should I forgive my brother?” is not improved by a rating scale. “Is this perfume beautiful?” can be informed by blind comparisons, but beauty is not a hidden quantity waiting to be measured. “What does fairness require here?” is a question whose difficulty is the point.

Established framework The philosopher W. B. Gallie described some concepts — he discussed examples such as democracy and art — as “essentially contested”: their meaning is part of what is being argued about, and the arguing is itself valuable. Forcing early precision on such words can end a conversation that should continue.

Interpretation Here the refining move is different. Instead of making the question measurable, make it articulate: which values are in tension, whose interests are affected, what facts are relevant even though they cannot decide, and what you would not do even to get the outcome you want. The refined question for “Should I forgive my brother?” might be: What would forgiveness ask of me, what would it not require, and what do I need from him first, if anything? That is clearer, not smaller.

Numbers can also be inappropriate for ethical rather than conceptual reasons. Scoring a partner’s affection, logging a child’s “gratitude,” or quantifying an employee’s “attitude” can damage the thing measured and hand power to whoever holds the spreadsheet. Sometimes the best refinement of a question is the decision not to measure it.