Risk Aversion as a Humility Heuristic

Considering higher-order uncertainty

My grad seminar recently discussed

Richard Pettigrew

’s intriguing paper, ‘Should longtermists recommend hastening extinction rather than delaying it?’ In the paper, Pettigrew (1) argues that moral decision-making should be risk-averse, (2) assumes a particular formal gloss on how to best understand what this involves, and (3) observes that this could entail pro-extinctionism, given even tiny risks of dystopian futures (since they end up being weighted so heavily under the chosen formalism). In this post, I’ll argue for a very different way of thinking about risk aversion, which avoids the pro-extinction implications.

The Problem with Formal Risk Aversion

It’s a fun paper. I also think it helpfully demonstrates the risks of formal philosophy (when it takes the form of blindly running with a particular formalism that is in no way forced upon us as the uniquely correct way to formalize the underlying phenomenon of interest). As I previously put it:

Unlike Pettigrew, I take this to constitute a decisive objection to those risk-averse decision theories. Whatever “commonsense” intuitions motivate the move to risk-averse decision theories in the first place clearly do not support the particular systematization that leads to this crazy result.

Recall that risk-averse decision theories are motivated purely by intuitions about cases. They want to capture people’s sense that it’s reasonable to “play it safe” in many ordinary contexts. (Pettigrew gives an example of a COVID-era party organizer choosing between paying for better air filters (to reduce infection risk) or better catering. He suggests that even if the better food would maximize expected utility for every individual, some may be risk averse, and it seems the organizer then ought to defer to the risk-averse.)

The problem is that the pro-extinction implications of formal risk-aversion are deeply counterintuitive. Imagine asking a ghostly soul, prior to embodiment: “Are you sure you want to risk being born? Here are the odds for your various possible lives: a 1 in 10 million chance of dystopian misery; a 1 in 100,000 chance of utopian happiness, and otherwise a normal moderately-happy life.” If your formalism for risk-aversion implies that ordinary people would rather not exist given these odds, then something has gone wrong. Such a verdict is no part of ordinary risk-aversion.

Clearly we should reject that formalism. The question is what to replace it with. I previously suggested that instead of (increasingly) over-weighting bad states per se, we could assign diminishing marginal value to good states above a sufficient level. This avoids formal pressure towards preferring non-existence over an excellent (and strongly positive expected value) shot at a decent life, merely on grounds of its including a tiny risk of suffering miserably.

I now want to explore a different, more “pragmatic”, explanation of some common intuitions in this vicinity.

Odds-Sensitivity and Epistemic Humility

Suppose you’re offered a small treat (e.g. a candy bar), at the cost of a 1/X chance of instant death. Make 1/X just tiny enough that the offer has positive expected value. Still, you may nonetheless feel it is reasonable to decline the offer, as simply not worth the risk.

Pandora was in a gambling mood

One very natural way to accommodate this kind of intuition is as reflecting a heuristic that protects you against the risk of mistaking the odds. You’re probably not great at distinguishing tiny numbers. In most realistic cases, you won’t really be certain about the odds anyway. And shifting the odds could completely change the expected value verdict.

Instead of being certain in a single point-probability, you should probably distribute “meta-credences” over a pretty wide range of first-order probability estimates. Once you do so, the implicit risk of death may start to look uncomfortably high: significantly higher than the 1/X “explicit odds” you began with. In that case, it actually maximizes risk-neutral expected value to reject the bet.

Or even if the odds are (somehow) guaranteed, you might not be certain of your calculative abilities. If your evil demon bookie offers you a calculator to double-check, you might worry that they’ve manipulated the circuitry. And so on. It doesn’t take much meta-uncertainty to swamp the first-order calculations when tiny numbers are involved. Even a one in a million chance that you’re being duped could easily swamp the “1/X” in the explicit calculation. (A small treat isn’t worth a 1 in a million chance of instant death, after all.)

So, rather than explicitly engaging in expected value calculations at all, it often makes sense for us to rely on heuristics. Avoiding big risks when there’s an acceptable alternative is one such heuristic. But it is not one that motivates hastening the end of the world just so as to avoid a tiny chance of dystopia. (Extinction is not, I take it, an “acceptable” outcome in the ordinary sense. It may be a “lesser evil” worth choosing in extremis, but it’s not so clearly “acceptable” that a sensible heuristic would advise you to settle for it in the sorts of cases we’re considering here—where the positive expected value of continued existence vastly outweighs the negative.)

So that seems like a pretty strong reason to prefer the heuristic account (or something like it) over formal—and potentially world-ending—risk-aversion. (Philosophers can always quibble over which apparent “counterexamples” turn out to be acceptable in reflective equilibrium. But I think it’s especially important to get the literal end of the world cases right.)

General Lessons

(1) Beware of formalisms: though often useful, they can also easily mask unwarranted extrapolations of arbitrarily-chosen features or models that happen to fit some narrow class of motivating cases, but haven’t really been established as the right explanation. When they lead to crazy verdicts, it’s always worth going back over the assumptions and checking for any choice points that were inadequately motivated. Remember that motivating the underlying idea (“risk aversion”) is not the same as motivating any particular formalization of how to implement the intended idea.

(2) Beware of taking intuitions about cases at face value as reflecting fundamental principles (cf. utilitarian toolkit item #1: “keep hypotheticals at a distance”). They may just reflect sensible heuristic verdicts.

Given that ordinary thought involves far more reliance on heuristics than on first principles, it’s odd that philosophers are so quick to interpret ordinary thoughts and dispositions as reflecting fundamental principles. (Often, I think, the correct response to, “Wouldn’t it be weird to say X?” is just, “Yes, that would be weird to say! It may not be something that an agent in the situation could reasonably be confident of. Still could be true, though—consider the costs of the alternative views.”)