Why studies from China should be taken with a sack of salt

Nothing new in what follows. Actually the studies I cite below are all over five years old. Just that of late I've been in a discussion with a doctor (an MD) who's also an acupuncturist and the topic of publication bias came up. We both agree that Chinese studies are suspect. But then he makes the simplistic claim that studies in the West are also guilty of the same bias. True enough that those in English-speaking countries are also afflicted with publication bias. But just as corruption is more prevalent in some countries than in others so too is publication bias. There is a difference in degree. And this isn't just a prejudice. There is evidence showing that this is a bigger problem among Chinese studies.

For instance, a 1998 systematic review by Andrew Vickers et al. ("Do Certain Countries Produce Only Positive Results? A Systematic Review of Controlled Trials") showed that 100% of acupuncture studies from China, Taiwan, Hong Kong, Japan, Vietnam were positive while of those from English-speaking countries 60% from the UK, 53% from the USA, and 30% from Canada, Australia, and New Zealand were positive [Bausell p.170]. Furthermore,

when Vickers and his colleagues repeated the analyses with a much larger sample of trials involving treatments other than acupuncture (most of which involved conventional medical treatments), they basically came up with the same results: 98 percent of conventional Chinese trials produced positive results, as did 97 percent of Russian trials [Bausell p.169]

In their 1999 "Review of randomised controlled trials of traditional Chinese medicine" Jin Ling Tang et al. noted that besides reporting inadequacies, small trial sizes, limited use of blinding, inappropriate controls (use of a control treatment which in itself has not been shown to be efficacious), etc., there was a preponderance of positive findings among the RCTs which may be attributable to publication bias:

Most trials claimed that the tested treatments were effective, indicating that publication bias may be common; a funnel plot of the 49 trials of acupuncture in the treatment of stroke confirmed selective publication of positive trials in the area, suggesting that acupuncture may not be more effective than the control treatments.

Zhenglun Pan et al. in "Local Literature Bias in Genetic Epidemiology: An Empirical Evaluation of the Chinese Literature" have this to say:

Chinese studies typically suggest much stronger genetic effects than non-Chinese studies, and this may be even more prominent for the few studies that reach PubMed. Although Chinese studies are smaller than non-Chinese studies and thus even more underpowered, surprisingly half of them reach formal statistical significance for the evaluated gene-disease association. This exaggeration is seen across very diverse topics.

The larger genetic effects in Chinese studies are unlikely to reflect genuine heterogeneity in the effects of genetic risk factors across various "racial" descent populations. Heterogeneity due to ancestry should not have led always to larger effect sizes in all probed gene-disease associations. Therefore, the most likely explanation is publication bias against "negative" results or other selection biases in the chase for statistically significant findings. This explanation is further supported by our analysis of the expected number of statistically significant findings. Even if the average genetic effects in the Chinese studies were indeed as large as those observed, one would expect far fewer Chinese studies to have reached formal statistical significance on their own, given their small sample sizes. The alternative explanation that Chinese investigators may be targeting high-risk populations with particularly strong genetic effects is unlikely given these data.

In his November 22, 2005 PLoS article "Selection Bias in Meta-Analyses of Gene-Disease Associations" JL Tang comments on the above study by Pan et al:

In their study published in this issue of PLoS Medicine, Pan and colleagues compared genetic studies conducted in mainland China with those from other places. The researchers identified 12 gene-disease associations and compared a total of 161 Chinese studies and 309 non-Chinese studies. The Chinese studies were on average smaller in sample size than non-Chinese studies and appeared in the literature a few years after the first non-Chinese studies. Chinese studies in general reported a stronger gene-disease association and more frequently a statistically significant result. These two characteristics were more likely to occur in Chinese studies identified through PubMed than in those accessible only locally.

These findings suggest a variation or heterogeneity in the strength of the gene-disease association (often expressed in an odds ratio) observed between Chinese and non-Chinese studies. These studies are primarily case-control studies. Many factors may contribute to the variation in the estimate of odds ratio across such studies, such as the genetic make-up of the population studied, the type of patients included, the selection of controls, the quality of the study design, and the quality of the laboratory work. These factors could lead to either over- or under-estimation of the true odds ratio. However, it is difficult to conceive that any single factor, or combination of these factors, could consistently cause the exaggerated odds ratio in Chinese studies in all the topics (gene-disease associations) examined by Pan and colleagues. Selective publication is therefore a very likely and worrying explanation for their findings.

Selective publication can cause publication bias, which in turn could lead to false gene-disease associations in meta-analyses. It would be a disaster if a genetic screening program (in which healthy people are tested for a gene and offered a treatment if they test positive) were based on such a false association. Even if such a false gene-disease association were only subjected to further related investigations, this would be a waste of valuable resources for medical research.

Selective publication of positive studies in China and a few other Asian countries has been observed in clinical trials of acupuncture. However, selective publication by no means exists in only the Chinese literature. It is probably a common phenomenon in the entire field of biomedical research. Given the fact that positive studies are more likely to be published than negative ones, and given the pressure on researchers worldwide to publish in indexed journals (especially in international journals with high impact factors), selective publication is likely to continue in the foreseeable future. As compared with English-speaking countries, selective publication is perhaps more likely to occur in non-English-speaking countries where there are a small number of indexed journals to publish local studies.

That last line bears repeating: "As compared with English-speaking countries, selective publication is perhaps more likely to occur in non-English-speaking countries where there are a small number of indexed journals to publish local studies."



----

Reference:

R. Barker Bausell. Snake Oil Science: The Truth About Complementary and Alternative Medicine. New York: Oxford University Press, 2007.