The Mismeasure of War

Our human rights team cares deeply about the issue of truth in human rights reporting. They just posted a response to a major debate on the question of whether deaths in war have been steadily declining. I think this well captures our approach to the question of truth and are worth repeating in the Beneblog.

A guest Beneblog by By Anita Gohdes, Megan Price, and Patrick Ball

Several media organizations including Reuters, Foreign Policy and New Scientist covered the January 21 release of the 2009 Human Security Report (HSR) entitled, “The Shrinking Cost of War.” The main thesis of the HRS authors, Andrew Mack et al, is that “nationwide mortality rates actually fall during most wars” and that “today’s wars rarely kill enough people to reverse the decline in peacetime mortality that has been underway in the developing world for more than 30 years.” This claim is based in large part on the authors’ graphical representations of pre- and post-conflict mortality rates for a variety of countries, and on their critique of five surveys conducted by the International Rescue Committee (IRC) in the Democratic Republic of Congo between 2000 and 2007.

The authors’ illustration of the technical mistakes made by the IRC is necessary and valuable for scientific advancement.

We fully agree with their assessment that some of the IRC extrapolations are inappropriate, that IRC should have calculated a population-weighted mortality estimate and that the estimates contain very high levels of uncertainty. Each of these concerns suggests a corrective process by which better estimates could be made. In their argument, however, the HSR authors did not choose this course.

We are deeply skeptical of the methods and data that the authors use to conclude that conflict-related deaths are decreasing. We are equally concerned about the implications of the authors’ conclusions and recommendations with respect to the current academic discussion on how to count deaths in conflict situations. See Andrew Gelman’s blog and the HSR’s own discussion site for an overview of this discussion. We believe that the authors should examine their own data on mortality related deaths with the same rigor with which they critique the recent IRC surveys.

If they did this, they would find that they have inadequate information to conclude anything about the trend in war-related lethality in recent decades.

The central evidence that the authors provide for “The Shrinking Cost of War” is delivered as a series of graphs. There are two problems with the authors’ reasoning.

First, the mortality estimates of children under five described in Figure 2.1 of the report should include an appropriate measure of uncertainty. The purported trend could be overwhelmed by the error of the estimates describing child mortality, but these errors are not presented in the report. Therefore, it is impossible to test the strength of the trend versus the magnitude of the error.

Second, and even more importantly, the graphs showing a worldwide decline in war-related lethality, in Figures 2.5 and 2.6 of the report, include data from the PRIO Center for the Study of Civil War, International Peace Research Institute, Oslo and the UCD/HSRP Uppsala Conflict Data Program /Human Security Report Project Dataset, the World Bank, “World Development Indicators” and the Inter-Agency Child Mortality Estimation Group (IACMEG), “Child Mortality Estimates Info.”

These datasets include expert opinions, convenience datasets, and reproducible estimates from multiple-systems estimation and probability-based surveys. The quality and uncertainty of such an incompatible collection of datasets cannot be evaluated. The plausibility of the trends presented by the authors cannot be assessed.

In particular, expert opinions about the magnitude of violence are little more than guesses. These numbers are the statistical equivalent of hearsay. It is impossible to scientifically debate speculations, and it is impossible to reproduce speculations by a principled process in alignment with the scientific method.

Some of the other sources for the authors’ conclusions are convenience samples. A convenience sample is simply data that can be conveniently observed via witness accounts, press sources, or other means. These databases may be useful collections of cases, and organizing information in this form enables many non-statistical descriptions of violence.

However, convenience samples are highly unlikely to represent the underlying statistical patterns or magnitude of conflict related deaths. They cannot be used to extrapolate to a population beyond what was observed, and they are not necessarily representative of any population. In our experience, no two convenience samples about the same country tell the same statistical story.

The core problem with both expert opinions and convenience samples is that we have no way of scientifically measuring just how unrepresentative they actually are of the population they are attempting to measure.

The HSR’s broadest claim is that “[t]he average conflict in the new millennium kills 90 percent fewer people each year than did the average conflict in the 1950s (p. 2),” and that there has been a “20-year decline in conflict numbers” (p. 7). Due to the weaknesses in the data, there is no way of reproducibly or transparently verifying or testing this claim.

In the HSR and elsewhere, the HSR authors have shown that recent conflicts have been seriously mismeasured. Certainly this debate would benefit from a scrutiny of expert opinions and convenience samples as intense as that which the HSR authors’ have brought to the study of the IRC surveys.

Perhaps earlier conflicts were as mismeasured as recent conflicts. The HSR authors’ conclusion that the number of deaths in today’s wars is declining may be right, or it might be wrong. But we just do not know much about the quality of estimates from further back in history. Therefore, the only responsible conclusion is that we simply don’t know what the trend in war-related deaths looks like. More rigorous research is needed.We welcome the authors’ contribution to the ongoing debate about measuring war-related mortality. Technical critique of existing work is at the core of the scientific process. In our opinion, the HSR authors have done the IRC and the community of human rights analysts a service by highlighting errors in the Congo survey.

The response to errors with statistical estimates must not be that we abandon science by relying on expert opinions and convenience samples. Quite the opposite. The fact that the IRC’s work has been shown by the HSR authors to be flawed should remind us to limit our conclusions narrowly to what can be defended by the most appropriate and advanced scientific methods for the question at hand.