Clinical evidence is fortunately no longer based on anecdotal case series. One design in particular that is emerging with the rise of so-called evidence-based medicine is the meta-analysis. It is significant that meta-analysis is the most widely cited design in biomedical literature, and it is thought to provide the highest level of scientific evidence, even exceeding that of the clinical trial.1 Meta-analysis is defined as a review technique that is used to systematically identify, assess, and pool data from all studies performed on the same topic, generally examining the association between an exposure factor and a disease. One area of particular interest to pulmonologists might be, for example, the relationship between contact with pets and the risk of asthma.2 Until recently, training in this technique represented just a small part of postgraduate epidemiology programs in Spain, but meta-analysis is now included in the training of all healthcare professionals. However, this basic education does not make a clinician an expert in meta-analytical methodology. It is striking, then, that some clinical colleagues with no specific training in epidemiological, let alone meta-analytical methodology, bite off more than they can chew when asked by journal editors to review articles, and instead of contributing their clinical knowledge choose to make methodological criticisms, sometimes even vehemently. The following are examples of some of the most common misguided criticisms of meta-analyses that experts find so exasperating:

• 1

  “The meta-analysis is not registered.” For some years, certain journals have recommended that meta-analyses be registered at a preliminary stage in the PROSPERO registry. The aim seems to be to avoid duplication of similar studies. Registration is not obligatory, and many experts believe it to be counterproductive because it encourages authors to register meta-analyses “just in case”, even though they may never see the light of day. In doing so, they prevent other authors from carrying out the research. Registration, then, is not a sign of good quality.3

• 2

  “The meta-analysis shows heterogeneity.” Repeating this criticism like a mantra in every review does not make it true. Heterogeneity, or observing different results in different subgroups, such as men and women, is not a defect of the meta-analysis but a characteristic that must be described and explored. To condemn a meta-analysis because its results are heterogeneous is equivalent to discarding an epidemiological study because the results are not statistically significant. Higgins, the greatest expert in heterogeneity, said that “heterogeneity is to be expected in a meta-analysis: it would be surprising if multiple studies, performed by different teams in different places with different methods, all ended up estimating the same underlying parameter”.4

• 3

  “A meta-analysis cannot mix case-control studies, which present odds ratios, with cohort studies, which present incidence rate ratios.” This error is common even among professionals with a certain level of epidemiological sophistication. The odds ratio is an unbiased estimate of incidence rate ratios in case-control studies, if the choice of controls in those studies was appropriate.5 Although rarity is not a prerequisite, the rare disease assumption is needed to ensure that the odds ratio is a good estimator of the relative risk, also known as the cumulative incidence ratio. The rate ratio and the relative risk are distinct entities that must not be confused. It is totally legitimate and desirable to combine cohort studies and case-control studies in an initial analysis.

• 4

  “The subgroup analysis was not predetermined before the start of the meta-analysis.” Some reviewers believe that results can only be stratified by subgroups if this procedure was decided before the start of the meta-analysis. In other words, no analysis can be done after seeing preliminary results. I wonder why a discovery emerging from preliminary results is less valid than one that arises from an analysis specified in the initial protocol. In a meta-analysis, we showed that contact with pets is an important risk factor for asthma in countries where, for cultural or religious reasons, pets are not usually kept in the home.2 The underlying idea was that frequent contact with some allergens could induce greater tolerance, while limited contact with pets could lead to less tolerance among subjects. There is no problem in the fact that we came to this conclusion only after observing that some of these studies had been carried out in Arab and African countries, where pets are not generally welcome indoors.

• 5

  “The meta-analysis lacks interest because it does not show any risk factor effect.” Showing an absence of effect is just as interesting as showing a presence of effect. The publication of a meta-analysis should not depend on its final result. Rejecting a meta-analysis because its results are negative is to confine scientific journals to a sensationalist role. The most useful meta-analysis among those published by my group is one which, despite the alarm in the media, showed that hair dyes had no effect on cancer risk.6

I hope these few examples from my own personal experience will raise awareness among my medical colleagues of the harm that can be done by a poor manuscript review. With the proliferation of on-line journals, any doctor who has ever published an article will sooner or later be asked to peer-review studies, some of which will be meta-analyses. It is clear that if research is to benefit from the experience of clinicians, it needs to descend from the ivory tower of academia, but a request for a peer review is not the equivalent of a blank check, and reviewers would be wise to limit their comments and criticisms to their area of expertise. In the words of Rousseau: “always remember that one does not lose one's way by what one does not know, but by what one believes he knows”. Before committing any of the errors mentioned above, let us be guided by common sense and apply the universal principle: in dubiis, abstine (if in doubt, abstain).