(See part 1 and part 2 and part 3.)
The critics of the Lancet study have had just one main argument against it: incredulity. They are like the intelligent design creationists who, because they cannot imagine that life as we know it could have evolved, simply assume that a creator must exist without even looking at the evidence.

Some try and make the case that if the level of deaths are really so high, the media would have reported it. The authors of the study are not idiots. They have considered this question in the light of what we know from other conflicts.

[The figure of 655,000] is far greater than reported by various media accounts and morgue tallies. This is not surprising, as reporting of events from incomplete sources cannot, in any statistically meaningful way, be converted into national death rates. Other than Bosnia, we are unable to find any major historical instances where passive surveillance methods (such as morgue and media reports) identify more than 20% of the deaths which were found through population-based survey methods.

Richard Horton, editor of Lancet writes:

[T]he reason for the discrepancy between these lower estimates and the new figure of 650,000 deaths lies in the way the number is sought. Passive surveillance, the most common method used to estimate numbers of civilian deaths, will always underestimate the total number of casualties. We know this from past wars and conflict zones, where the estimates have been too low by a factor of 10 or even 20.

In the comments to yesterday’s post, Eldan Goldenberg refers to a critique of the Lancet study put out in a press release by IraqBodyCount. The main thrust of their comments is that for such a large number of deaths to go under the radar implies massive breakdown of the system. But the Lancet editor seems to be saying that that kind of official undercounting is the norm is such situations, not the exception. It is just that this feature is not highlighted in other situations. Again, the IraqBodyCount critique is not of the study itself but based on the feeling that the figure is ‘unreasonable.’ But with research, we cannot adjust figures based on the reasonableness. All that surprising results require of their authors is careful scrutiny of the methodology to see if systematic errors have distorted the results. Researchers do not have the luxury of adjusting figures

However, the authors of the Lancet study, like any serious scholars, realize that their study has limitations and reflect on them and their possible effects.

Any collection of information is open to potential bias, and has limitations. All efforts were made to randomly select the households to be included in this survey, but it may have been that households with more deaths or households with fewer deaths were over represented in this survey. The finding that the 2006 results are very close to the 2004 household results suggests this did not occur. As in all surveys, a larger sample would have likely have produced a result with greater precision, although this would have exposed the survey teams to higher risk. In the future, when safety has improved, a large survey will be needed to determine in detail the total implications of the conflict for the people of Iraq.

The households were selected for this survey according to population size we obtained from the Ministry of Planning, but this may not have fully reflected migration within or outside the country. However, it is unlikely that this would have occurred at a scale necessary to affect findings.

Perhaps the greatest potential limitation to this type of survey is the problem people have recalling the date of specific events, especially over several years. Again, the close similarities between the 2004 and the 2006 data suggest this was not a major problem. Households could have concealed deaths from the interviewers, though by promising anonymity to households we tried to minimize this risk. We are certain that households did not report deaths which did not occur, as 92% of households had death certificates for deaths they reported.

Another reason to have confidence in this study lies in the very process of peer review. When papers are submitted to scientific journals, the referees cannot and do not verify the actual data. What they look to see is whether the study has followed good methods and the authors have explored all reasonable alternative explanations before reaching their conclusions. This is especially done when the results are so surprising, as in this case. The editors of Lancet, clearly mindful of the explosive political nature of this paper, sent it to four referees and you can be sure that those referees checked to make sure proper procedures were followed. This does not mean that the results could not be wrong. Peer review has failed in the past to detect errors and is not designed to detect outright fraud. But it does mean that peer-reviewed papers have prima facie credibility and if you want to challenge their veracity, the burden shifts to you to do so. If you want to discredit it, you have to produce contrary data or detect a serious flaw in the methodology, or show that there has been an error in the calculation. I have not seen any criticism along these lines as yet.

The authors of the study also describe how the data was collected.

The two survey teams consisted of two females and two males each with one male supervisor. All were medical doctors with previous survey and community medicine experience and were fluent in English and Arabic. All were Iraqis.

Those of us who wonder how such studies based on surveys can be carried out within a war zone have to give credit to the courage and dedication of the people who did this. I know that people who try to collect accurate information in war zones run great personal risks because what warring factions want to avoid most is any accountability and they resist efforts by people to collect data. This is why fact-finders in conflict zone deserve our greatest respect and admiration. These ten brave Iraqis did not want their names revealed for fear of retribution. The fact that the ten Iraqi doctors were willing to risk their lives to try and get information about their ravaged country did not want to be identified testifies to the dangerous situation they were in and I for one share the sentiments of the study authors when they write: “We express our deepest admiration for the dedicated Iraqi data collectors.”

POST SCRIPT: Dawkins and Colbert-What could be better?

Watch a highly entertaining interview of Richard Dawkins by Stephen Colbert about Dawkins’ new book The God Delusion