Evaluating Philosophy
Can we do better?
Academics often gripe about “referee 2”. A larger problem, at least in philosophy, is that we just aren’t very good at assessing each other’s work. But we could do better: there are particular aspects of our academic culture that feed into bad evaluative practices. Attention to these issues (perhaps via better directions from journal editors to their referees) could lead to systematically better-warranted verdicts. Or so I hope.
So, what’s the problem? Some may stem from ineliminable disagreements about what constitutes good philosophy. But for now I want to focus on a more tractable issue: mistakenly prioritizing the not bad over the actually good.
Seeking Good Philosophy
As I’ve argued elsewhere, people tend to focus overly much on avoiding bads relative to attaining goods. This can lead to evaluating papers according to an implicit norm of objection-minimization. Taken to extremes, such an approach implies that a blank sheet of paper is the ideal work of philosophy. Any words added just increase the risk of flaws. The job of a referee is to hunt out those flaws, and once sufficient flaws have been identified, recommend that the paper be rejected. (Often editors will encourage this approach by reminding referees that the journal has a large backlog and would appreciate their referees using “especially high standards”—which, in the absence of the sorts of clarifications I’ll recommend below, ends up implicitly communicating: “Look for any objection as a basis to reject!”) If no objections spring to mind, the paper may be accepted: it’s as pure as the blank page from which it sprang.
The problem with this approach is that it treats being unobjectionable as the highest philosophical virtue, when it is really closer to a vice. Interesting philosophical work is rarely unobjectionable. The main way to be unobjectionable in philosophy is to not say anything of interest. And, indeed, plenty of papers succeed at this task. But we shouldn’t encourage it.
Think of the most interesting philosophy papers you’ve read: fascinating work by David Lewis, Derek Parfit, David Chalmers, Peter Singer, Judy Thomson, Caspar Hare, the list goes on… Absolutely all of it is subject to reasonable objections, because interesting philosophy is contestable. So any referee who takes the presence of “reasonable objections” to be sufficient grounds for rejecting a paper is implicitly committed to depriving us of any interesting philosophy in future.
Objections are good grounds to reject a paper if they show that the argument is actually rather hopeless, such that no well-informed person should find it persuasive. Transparently bad arguments are certainly of little interest. But again, it’s irresponsible to reject a genuinely interesting and informative paper simply because you happen to reasonably disagree with some of its premises. The philosophical interest of the conditional knowledge contained in the paper—i.e., about the surprising implications of the premises—does not depend upon actually accepting the premises. The potential interest of the implications instead depends upon the premises being such that they could reasonably appeal to some (ideally: many) who don’t antecedently accept the conclusion. Clarifying a genuinely appealing area of logical space is philosophically valuable, even if the evaluator is personally more drawn to a different location.
So instead of chasing objections, referees should primarily aim to evaluate whether a paper sufficiently possesses the philosophical virtues that are found in the best works of philosophy: whether it is clear, original, ambitious, significant, thought-provoking (perhaps opening new avenues of inquiry), clever, and apt to rationally persuade some well-informed fence-sitters (the more, the better). All of that is compatible with its also being such that some readers—including the referee—could have reasonable grounds to reject one or another of the premises.
Good-Seeking Questions
Supposing I’m right about the above, how can we go about improving philosophical evaluation in practice? I think the following questions are all good ones for a referee to ask themselves (and even better questions for editors to ask of their referees):
(1) What (if anything) is interesting and original about this paper? How interesting is it? (How much light does it shed on previously-shadowy areas of logical space? How philosophically important is the illuminated content?)
(2) Are there any egregious errors or oversights that would need to be addressed before the paper was potentially publishable?
(3a) How cogent are the paper’s central arguments?
(3b) Do you expect most other experts would share your verdict, or is there significant room for reasonable disagreement here?
(3c) How likely is it that a reasonable, well-informed (but currently agnostic) spectator of the debate would shift their credences as a result of reading this paper?
(4) Compared to other papers recently published in this journal, is this paper (vastly or slightly) more or less likely to be (a) widely cited, (b) assigned in graduate seminars, (c) significantly reshape the debate, and (d) still read or remembered a century hence?1
→ Would you add or change anything in the above list?
With some wonderful exceptions, referee reports in philosophy tend to systematically overweight (often idiosyncratic) judgments about whether the argument is successful (3a). Questions like 3b & 3c strike me as much better for purposes of objective professional evaluation. And questions (1) and (4) strike me as most important to the overall evaluation that a paper warrants.2
Note that scoring well by the criteria in (4) seems sufficient for warranting publication in a top journal. Whatever complaints a referee might have about a paper, they obviously shouldn’t deprive the discipline of a paper that could prove so significant and widely valued. Those are clearly the papers that journals should want the most.
You might even get away with just asking (4), since the answers would presumably have to take into account all the other factors. But maybe it is helpful to explicitly step through the other questions along the way (especially for giving authors illuminating feedback).
It might save time to note that scoring poorly by criterion (1) seems like a sufficient basis for rejecting a paper, at least in top journals. Decisive proofs of trivialities aren’t worth taking up journal space, or readers’ time.
Conclusion
A nice structural feature of these good-seeking standards is that they encourage doing more rather than less philosophy in a paper. (It’s always disappointing to read an 8,000 word paper and think, “This would have been better as an 800-word blog post.”)
Given current norms, we all know that it can make a paper “more publishable” (i.e. referee-proof) to remove interesting ideas from it, because more content just creates more of a target for referees to object to. This is messed up. Good-seeking standards instead recognize that adding relevant valuable content is (typically) a good thing. Our evaluative standards should reflect this fact.
Most importantly, our evaluative standards should help us to identify good work, which objection-minimizing standards are very bad at.
Helen De Cruz recently noted that “Being unsure, reviewers tend to err on the side of caution and rather reject something that could be interesting than accept something that doesn’t work.” But recall that unreflective “caution” can easily constitute a kind of incaution. Good academic work is rare and valuable: mistakenly burying an objectively good paper is a serious loss to the discipline—a harm we should strive to avoid. Mediocre academic work is commonplace and largely harmless: mistakenly accepting a paper that objectively “doesn’t work” is no big deal. So it’s actually mistaken rejections, not mistaken acceptances, that constitute the graver risk here. (Not least because the bad incentives might result in less interesting work being written in the first place.) Focusing more on good-seeking questions, like those proposed above, may help to reduce that risk.
I welcome further suggestions / refinements of good-seeking questions to better elicit more accurate evaluations of philosophical work.3
It may also be worth explicitly trying to correct for the influence of momentary fads, since philosophical fashions come and go. Many philosophers seem keen to chase the latest “hot topic”, whereas I think we’d generally do better to ignore this and aim for more timeless (work and) evaluations.
A possible downside is that they’re especially “subjective” judgment-calls. But I think there’s no avoiding the fact that academic evaluation requires good judgment. If the evaluators lack this trait, then the whole endeavor seems rather hopeless.
As is so often the case, I’m especially indebted to Helen Yetter-Chappell for many conversations over the years that have influenced my views here.
Great post. Fortunately, some journals do explicitly seek these goods. From the Journal of the APA's instructions to referees:
"The Journal aims to publish papers only of the highest quality. This means that we will reject many very good papers. You should bear this in mind when you review a submission. Recommend papers only when they stand out. In general, we hope to avoid papers that
• embed an interesting point in a lengthy discussion of the literature or argue preemptively against every possible objection;
• simply add an epicycle to a well-known thesis;
• are needlessly technical or introduce gratuitous formalism;
• are accessible only to specialists steeped in the topic.
The editors have a preference for papers that go out on a limb, that exhibit daring, that challenge the status quo, papers that defend surprising conclusions, even when the author’s arguments are not watertight or otherwise impervious to criticism. We rely heavily on reviewers’ judgment and good sense in this regard."
Honestly, "the pursuit of the unobjectionable" is the main reason I've tired of sending work to journals. And since I'm not interested in writing books, it basically means that I'm not publishing much -- which, given that I'm a full professor, is fine. But still: I wish it were otherwise.
I once received a referee report from Phil Studies that said the main problem with the view presented in my paper was that it was obviously false, but that it was "a testament to the undoubted philosophical skills of the author" that I was "able to make such an interesting case for an obviously false view." The editor rejected my paper on this basis. It was the first paper I submitted for publication as a young tenure-track prof, and I never sent it out again. (A version of the paper later appeared in an edited volume.)