What Would You Do Differently If You Knew?

by Mardoqueo Arteaga

TL;DR: More information can improve our understanding without improving the decision in front of us. I built a simple model to explore when another piece of research is worth its cost. The answer depends on the consequences of being wrong, the quality of the evidence, and what we can learn afterward. One particularly interesting result: a report that would be useless as the last piece of research can be valuable as the first.

There is a question I think we should ask more often before commissioning another analysis: what would we do differently if we knew the answer? It sounds almost too simple. We are used to asking whether the data are reliable, whether the sample is large enough, or whether the result holds under a different specification. All good questions, and ones I have spent a fair amount of time asking myself. But we can answer each of them carefully and still produce something that does very little for the decision it was supposed to inform.

I have been thinking about this as I move from LinkedIn to an exciting new role. Much of my recent work concerned what an intervention changed relative to a counterfactual. A question that interests me now is how we decide which uncertainty is worth resolving before taking an action. There is a connection to the themes I have been exploring here: how beliefs are formed, what information makes it into a decision, and why more observable activity need not mean more useful understanding. This time, I wanted to see what a simple model could tell us about when to stop gathering information.

Suppose a business is considering a launch. If demand is strong, it earns $100,000; if demand is weak, it loses $100,000. These are net payoffs, after the costs of launching. Declining the opportunity produces zero. For now, the business thinks each state is equally likely, so launching has an expected payoff of zero too. It is sitting exactly at the point where a little evidence could change its choice.

The business can buy a research signal for $2,000. Imagine a tightly specified customer study that returns either a favorable or an unfavorable result. When demand is strong, it returns a favorable result 70 percent of the time. When demand is weak, it returns an unfavorable result 70 percent of the time. We know those probabilities in the model. In practice, estimating them would be a substantial part of the work.

A favorable result raises the probability of strong demand from 50 to 70 percent. Launching then has an expected payoff of $40,000. An unfavorable result lowers that probability to 30 percent, at which point the business declines. Either signal is equally likely before the study, so the expected value of studying and then choosing is half of $40,000, minus the $2,000 research cost: $18,000.

That is a fairly good return on an answer. Notice, though, what creates the value. The firm uses the favorable result to launch and the unfavorable result to avoid launching. It gets to make its action contingent on what it learns. The $18,000 is an expected improvement across possible outcomes, not a promise about what happens on any particular launch.

Same report, different answer.

Now suppose the business begins with only a 20 percent probability of strong demand. The same favorable signal raises that probability to about 36.8 percent. That is a meaningful revision in its understanding of the opportunity. It is still insufficient to make launching attractive. An unfavorable result also leaves it declining. If this is the final piece of research available, neither result changes the action, and paying for it reduces expected value by $2,000.

I find this distinction useful because it separates learning something from gaining something through a particular decision. The report is informative. It does not improve the final choice when this is the only study available. That conclusion also relies on our deliberately narrow menu: launch at the stated scale or decline. If the firm could resize the launch, change its price, or use the findings for another product, the same evidence might become valuable. Information is being valued relative to the choices it can inform.

But there is another complication. What if that report is the beginning of a research program?

Starting from 20 percent, two favorable signals in succession raise the probability of strong demand to about 57.6 percent. Evidence that could not change the action after one study can change it after two. And the business need not commit to buying both studies. It can buy the first, see what it learns, and decide whether the second remains worthwhile.

This is where the sequence matters. A rule that asks whether the next report alone will change the action can stop too early. We need to value the next report together with the opportunities to learn that follow it.

Working backwards to know when to stop

To explore this, I built a model in which a business can launch, decline, or buy another study. It can purchase up to eight studies, updating its beliefs after each result and deciding whether to continue. Each study has the same cost and reliability, and contributes fresh evidence rather than repeating what the business already knows.

The method is called backward induction: start with the decision at the deadline and work backward, comparing the value of acting now with the value of learning more and deciding afterward. It is Bayesian because beliefs change with the evidence. That is the basic algorithm, though I am thinking about complementing these posts with more technical notes for all you interested parties. The point is to account for the possibility that an answer changes what it is worth asking next. This follows an established approach to sequential decision-making, applied here to an illustrative business problem.

The chart shows what happens as we change the business's initial belief about demand. The horizontal axis is its starting probability of strong demand. The vertical axis measures how much the opportunity to research adds to expected value, after research costs, compared with deciding immediately. The orange line allows at most one study; the teal line allows up to eight, with the option to stop after any result.

Research adds the most value near the middle, where the business is closest to changing its decision. Near either end, its beliefs are sufficiently strong that the available research is not worth buying. The teal line extends farther in both directions: having room to follow up can make investigation worthwhile even when a single study would not change the final choice. These are hypothetical expected values, not observed business returns, though you can imagine a scenario where you’ve been in a similar enough boat.

At 20 percent, the opportunity to investigate sequentially is worth about $3,754. The model calls for an average of roughly 2.43 studies. Starting at 50 percent, the opportunity to research adds about $27,379 and uses about 4.82 studies on average. A budget for eight studies need not become a plan to conduct eight studies. The business stops when the results justify stopping, including when the evidence becomes sufficiently discouraging.

The symmetry on either side of the chart comes from our equal gains and losses and symmetric signal quality. It is a feature of these assumptions, not a claim about actual business opportunities (again, you may be in a boat where you have similarities to this; I know I have). Increase the downside relative to the upside and the probability required to justify an immediate launch rises. Increase the cost of research and fewer learning opportunities are worth pursuing. None of these thresholds is a universal standard for how confident someone ought to be.

So what are we paying to learn?

The practical implication I take from this is that a research brief needs an account of the decision. What are the available actions? What would make us switch between them? What evidence could move our beliefs enough to matter, either immediately or through a feasible sequence of follow-up work? Without those answers, we can spend a great deal of effort improving the precision of a number whose value to the business remains unspecified.

This also helps distinguish a larger information supply from a better one. Five reports may share the same underlying source. Our model assumes that, for a given demand state, each study provides independent evidence. Feeding it five retellings of one source as five independent observations would manufacture confidence. A cheaper summary can make existing evidence easier to use, which is valuable in its own right, but it does not automatically constitute another independent observation. The distinction matters whenever we count the volume of research as a measure of how much we have learned.

There are important omissions here, which are probably worth mentioning. Demand never changes while the firm studies it, and the value of waiting is captured only through a constant cost per signal. Competitors do not react (generally untrue), the firm judges opportunities by their average monetary payoff, and information has no use beyond this one launch (which is also untrue since products often build off each other). A richer model could allow a shrinking opportunity window, a small pilot, uncertain signal quality, or learning that carries over to future decisions, which would effectively change the stopping rule. The calculation provided is a way to make these assumptions visible, rather than evidence that any real research team should stop at a particular probability.

Still, I think this thought exercise gives us a useful question to carry into the next meeting. Before asking for another cut of the data, describe how the answer could change the decision, or the next worthwhile question. Sometimes that explanation will justify more research than we first expected. Sometimes it will tell us that we already know enough to choose. Either way, the point at which we stop should have an economic explanation.

Notes and further reading:

[1] Thomas J. Sargent and John Stachurski, “A Bayesian Formulation of Friedman and Wald’s Problem,” QuantEcon. Background on Bayesian updating and sequential choice with sampling costs. The finite-horizon launch/decline example here is a separately implemented illustration, not a reproduction of their numerical example.

[2] MIT OpenCourseWare, “Expected Value of Perfect Information,” IDS.333 Risk and Decision Analysis. Further background on valuing information through decisions.