Update - Chapter 2

Information Retrieval: A Health & Biomedical Perspective (Second Edition)

Update to Chapter 2 - Health and Biomedical Information

2.1 What is information?

2.2 Theories of Information

2.2.1 Shannon and Weaver

2.2.2 Other models of information

(4/6/04) Another model of information is as "bundles," which consist of small, annotated, and organized units that humans employ as "track" through information space (Gorman et al., 2000). The methods to observe their use and guide development of digital libraries has been described by Gorman et al. (2002). Ash et al. (2001) have described how bundles appear in clinical settings, from formal information instruments such as flowsheets to scribbled annotations on the wrappers of gauze pads.

Gorman, P., Ash, J., Lavelle, M., Lyman, J., Delcambre, L. and Maier, D. (2000). Bundles in the wild: managing information to solve problems and maintain situation awareness. Library Trends, 49: 266-289.
Gorman, P., Lavelle, M., Delcambre, L. and Maier, D. (2002). Following experts at work in their own information spaces: using observational methods to develop tools for the digital library. Journal of the American Society for Information Science & Technology, 53: 1245-1250.
Ash, J., Gorman, P., Lavelle, M., Lyman, J., Delcambre, L., Maier, D., Bowers, S. and Weaver, M. (2002). Bundles: meeting clinical information needs. Journal of the Medical Library Association, 89: 294-296.

2.3 Properties of scientific information

2.3.1 Growth

(4/5/03) Another means to assess the growth of medical knowledge has been to measure the weight of the paper volumes of Index Medicus (the paper-based forerunner to MEDLINE). Durack (1978) found that Index Medicus remained relatively steady in weight from its inception in 1879 through the mid-1940s at approximately 2.0 kg. After that point, however, it began to increase in weight exponentially, growing nearly sevenfold in size between 1955 and 1978. Durack noted that this growth aproximately followed de Solla Price's observations about the "doubling time" of scientific literature. He expressed concern that if this trend were to continue, Index Medicus would consist of 200 volumes and weigh 1,000 kg by 1985. (Interestingly, he did not think about the database in its electronic MEDLINE form and the much lighter weight of electronic data. In fact, the current version of MEDLINE, even with its 40 gigabytes of size, weights practically nothing on a set of DVD disks! But that's not really what he was measuring.)

Madlon-Kay (1989) extended the analysis for another decade. She found that the growth after 1978 reverted to a linear rate. She expressed relief that Index Medicus would not exceed 1,000 kg until the year 2027. (As if the paper version will still be published by then!)

Durack, D. (1978). The weight of medical knowledge. New England Journal of Medicine, 298: 773-775.
Madlon-Kay, D. (1989). The weight of medical knowledge: still gaining. New England Journal of Medicine, 321: 908.

2.3.2 Obsolescence

(3/1/03) Another study of information obsolescence looked at the "truth survival" of conclusions in the domain of liver cirrhosis and hepatitis (Poynard et al., 2002). The goal of the study was to determine whether information generated by the best evidence-based means had a longer survival when obtained in studies of higher methodological quality. The authors identified 474 conclusions in the published literature from 1945-1999 and found that 285 (60%) were still true in 2000, 91 (19%) were obsolete, and 98 (21%) were false. The half-life of truth in this domain was 45 years (in stark constrast to the half-life figures for citations presented in the book). The most interesting results were that survival of conclusions was not higher in studies of better methodological quality than those of lesser quality, and that the 20-year survival of conclusions derived from meta-analysis was lower (57%) than in non-randomized studies (87%)or randomized controlled trials (85%).

Poynard, T., Munteanu, M., et al. (2002). Truth survival in clinical research: an evidence-based requiem? Annals of Internal Medicine, 136: 888-895.

(4/4/05) An additional study about the long lead time between publication of scientific results and their adoption into clinical practice comes from Balas and Boren (2000), who noted an average of 17 years between something first being discovered and put into routine clinical practice. Many medical tests and treatments do get accepted into practice more quickly than this, but for some, the lag time is probably too long.

Balas, E. and Boren, S. (2000). Managing Clinical Knowledge for Health Care Improvement, 65-70, in vanBemmel, J. and McCray, A., eds. Yearbook of Medical Informatics 2000 - Patient-Centered Systems. Stuttgart, Germany. Schattauer.

2.3.3 Fragmentation

2.3.4 Linkage

(4/3/03) Another large-scale project of bibiographic linkage is the Erdös Number Project. This project is part of the "folklore of mathematicians," who measure their distance in co-authorship from the prolific Hungarian mathematician, Paul Erdös. Erdös published over 1,400 scientific papers and had over 500 co-author collaborators. The mathematical community has undertaken building a collaboration graph for its community with approximately 337,000 authors of 1.6 million authored items in the Math Review database. Erdös is at the center of that graph. An “Erdös number” is thus the smallest number of coauthorship links between an individual and Erdös. Therefore, someone who co-authored with Erdös has an Erdös number of 1. Anyone who co-authored with any one of those co-authors has an Erdös number of 2. I was surprised to find that I have a relatively low Erdös Number of 4, thanks to my former postdoc Andrew Turpin (http://goanna.cs.rmit.edu.au/~aht/), who was a graduate student of Alistair Moffat ( http://www.cs.mu.oz.au/~alistair/ ), who has one of the lowest Erdös numbers (2) in the IR community.

The Web site for the Erdös Number Project is at:
http://www.oakland.edu/~grossman/erdoshp.html

(4/3/03) A paper analyzing bibliometric issues in the ACM SIGIR community was published in 2002:
Smeaton, A., Keogh, G., et al. (2002). Analysis of papers from twenty-five years of SIGIR conferences: What have we been doing for the last quarter of a century? SIGIR Forum, 36(2): 39-43. http://www.acm.org/sigir/forum/F2002/smeaton.pdf.

(3/31/04) Garfield et al. (2003) have developed a tool called HistCite, which provides a historiographic depiction of the citations emanating to and from papers. They demonostrate that for seminal papers, the important literature of a field can be easily identified.

Garfield, E., Pudovkin, A. and Istomin, V. (2003). Mapping the output of topical searches in the Web of Knowledge and the case of Watson-Crick. Information Technology & Libraries, 22: 183-187.

2.3.4.1 Citations

(3/31/04) Another type of analysis done in bibliometrics is co-citation analysis, which measures the number of times that pairs of authors are cited together by another paper. Co-citation analysis can help show authors whose work is similar in scope. Andrews (2003) performed such an analysis for the field of medical informatics, with a particular focus on members of the American College of Medical Informatics (http://www.amia.org/acmi/acmi.html), a body of elected fellows who have made significant and sustained contributions to the field. This article shows not only that my work is closest to that of Keith Campbell, Betsy Humpreys, Mark Tuttle, and Christopher Chute, but that I am the 21^st most highly cited individual in this group of leaders of the field.

Andrews, J. (2003). An author co-citation analysis of medical informatics. Journal of the Medical Library Association, 91: 47-56.

(4/9/06) An additional analysis of publication and citation in the medical informatics field was recently carried out by Eggers et al. (2005). They analyzed ten years of publications and citations in 22 journals in the field. In addition to measuring numbers of publications, they calculated an "authority score," based on frequency of citation by other highly frequent publishers. I rank 10^th in the list of authority scores! They also map closeness of authors, which converges into five topic areas of the field. This anlaysis places me close to Homer Warner, Steve Johnson, Arthur Elstein, and Patricia Brennan, which actually seems less logical than the Andrews analysis above.

Eggers, S., Huang, Z., et al. (2005). Mapping Medical Informatics Research, 36-62, in Chen, H., Fuller, S., Friedman, C. and Hersh, W., eds. Medical Informatics: Knowledge Management and Data Mining in Biomedicine. New York, NY. Springer-Verlag.

(4/4/07) As noted in section 2.6.1 of the book, Lawrence (2001) found that journal articles from computer science were more likely to be cited if they were freely available on the Web. Antelman (2004) recently extended this analysis to four other fields (philosophy, political science, mathematics, and electrical engineering) and found that articles published under the open-access model (defined in section 2.6.1 below) had a statistically significant higher number of citations. Likewise, Eysenbach (2006) assessed articles published in Proceedings of the National Academy of Sciences, which offers both open-access and non-open-access publshing to authors, and found that those published under the former approach were 2-3 times more likely to be cited. Of course, these articles were not "randomized," so there may have been confounders leading to the different citation rate.

What other factors lead to higher rates of citations? A pair of studies has looked at Norwegian scientists, most of whom publish in international journals and are cited by international authors. The first study looked compared highly cited with "ordinary" cited papers (Aksnes, 2003). It found that while most papers were regular articles (81%), review articles (12%) were over-represented relative to their regular rate of appearance. Highly cited papers follow the usual pattern of rise and decline of citation frequency over time. They receive citations from many different journals and from both close and remote fields, which is also true of ordinary papers, although their high rate of citation makes them appear in higher absolute numbers in different journals and fields. The second study found that in general, the rate of citation of papers correlated well with scientists' perceived importance of the research (Aksnes, 2006). However, due to variance, it was difficult to apply this at the individual paper level.

Aksnes, D. (2003). Characteristics of highly cited papers. Research Evaluation, 12: 159-170.
Aksnes, D. (2006). Citation rates and perceptions of scientific contribution. Journal of the American Society for Information Science and Technology, 57: 169-187.
Antelman, K. (2004). Do open-access articles have a greater research impact? College and Research Libraries, 65: 372-382.
Eysenbach, G. (2006). Citation advantage of open access articles. PLoS Biology, 4(5): e157.
Lawrence, S. (2001). Free online availability substantially increases a paper's impact. Nature, 411: 521.

2.3.4.2 Author productivity - Lotka's Law

2.3.4.3 Subject dispersion - Bradford's Law

(3/31/04) Another area where Bradford's Law has been shown to apply is Web site citation analysis. Cui (1999) analyzed the Web citations (links) for library sites from 19 of the top 25 ranked medical schools in the US. The distribution of top-level domain (e.g., .com or .edu), first-level domain (e.g., the part of the URL up to the first slash, e.g., www.irbook.info/), and whole URLs were analyzed. When the total number of first-level domains were segregated into three groups based on total frequency (1731), their absolute counts came close to obeying the 1:n:n² distribution (78:452:1201 or 1:4:42). This study also found that 90% of the top-level domains were for the US-based top-level domains (.com, .edu, .gov, .org).

Cui, L. (1999). Rating health web sites using the principles of citation analysis: a bibliometric approach. Journal of Medical Internet Research, 1: e4. http://jmir.org/1999/1/e4.

2.3.4.4 Journal importance - Impact Factor

(4/1/03) Another study assessing the validity of impact factor (IF) was recently performed (Saha et al., 2003), asking 113 physician who were predominantly practitioners and 151 physicians who were graduates of advanced training programs in clinical and health services research to rate the quality of nine general medical journals. The correlation of IF and physicians' rating of journal quality was high overall (r² = 0.82), and higher for the group of researchers (r² = 0.83) than the practitioners (r² = 0.62). Table 2 in this paper provides a fascinating look at the subjects' rating of quality, the IFs, and the numbers who read and/or subscribe to the journals.

Saha, S., Saint, S., et al. (2003). Impact factor: a valid measure of journal quality? Journal of the Medical Library Association, 91: 42-46.

(4/5/03) A footnote to Table 2.1: In 2001, the IF for the Journal of the American Medical Informatics fell precipitously to under 1.0. The reason for this is that the Proceedings of the AMIA Annual Symposium, labeled as the "JAMIA Supplement," were included in the calculation. This vastly increased the formula's denominator far out of proportion to the numerator. As such, AMIA has removed the "JAMIA Supplement" moniker from the conference proceedings and the IF should return to its previous level.

(4/12/06) The debate over the true value of IFs continues and will probably never end. Nakayama et al. (2003) assessed the IFs of the citations included in the US government's Guide to Clinical Preventive Services, Second Edition. This guide reflects the best evidence for clinical preventive services. Not surprisingly, the largest number of citations came from journals with high IFs. Of the 1,740 citations in the 25 chapters of the report, the most commonly represented journals were Journal of the American Medical Association (135), American Journal of Preventive Medicine (102), British Medical Journal (77), and Lancet (70). The IFs of the 56 journals having five or more citations in the report were widely distributed, however. Six (11%) journals had an IF >10, but half of the journals (28, or 50%) had an IF < 3, and the median IF was 2.76. There was a correlation between IFs and number of times cited in these guidelines. However, this analysis showed that many articles having high-quality clinical evidence were published in low-IF journals. An editorial in British Medical Journal assessed some of the other social aspects of IFs, such as academic promotion, in light of these findings and advocated the dumping of IFs (Abbasi, 2004).

The inventor of the IF, Garfield, recently defended it in a JAMA commentary (Garfield, 2006). He noted the value of quantitative measurement of a journal's importance for decision-making concerning the value of journals as well as decisions for library subscriptions. He did, however, warn against its use for evaluating the contributions of indivudual scientists. Dong et al. (2005) also recently reviewed the IF. They provided a list of how the measure can be biased, including:

Incomplete journal coverage by the Science Citation Index (SCI), with an over-representation of English-language journals.
Different citation patterns across research fields and subject areas.
Differences among journals that increase IF but have nothing to do with quality, such as citations to "non-citable" items (i.e., items not counted in the IF denominator), availability of the abstract and/or full text on-line, and presence of longer articles.
Inaccuracy of data (e.g., the JAMIA example above)
IF calculated for whoe journals whereas citations are to single articles.
Journals manipulating IF, such as by requesting authors to cite papers in their journal.

They note that while IF is effective in measuring the quality of a journal, it has little value in assessing individual articles, whether for their scientific contribution or how evidence-based they are (e.g., the Nakayama paper above). They look askance at policies like those of the Finnish government that evaluate scientists by tracking "publication points" based on the IF of journals in which they publish (Adam, 2002).

Abbasi, K. (2004). Let's dump impact factors. British Medical Journal, 329. http://bmj.bmjjournals.com/cgi/content/full/329/7471/0-h.
Adam, D. (2002). The counting house. Nature, 415: 726-728.
Anonymous (1996). Guide to Clinical Preventive Services, Second Edition. Washington, DC. Office of Disease Prevention and Health Promotion, Department of Health & Human Services. http://odphp.osophs.dhhs.gov/pubs/GUIDECPS/.
Dong, P., Loh, M., et al. (2005). The "impact factor" revisited. Biomedical Digital Libraries, 2: 7. http://www.bio-diglib.com/content/2/1/7.
Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295: 90-93.
Nakayama, T., Fukuhara, S., et al. (2003). Comparison between impact factors and citations in evidence-based practice guidelines. Journal of the American Medical Association, 290: 755-756.

The most recent impact factors (IFs) from the categories of general and internal medicine, medical informatics, biotechnology (including bioinformatics), and computer science information systems (including information science) journals respectively are shown in the tables below from the 2004 edition of the ISI Journal Citations Report. Note the presence of JAMIA in three out of the four lists! Also note the high citation rates of bioinformatics journals generally.

General and Internal Medicine Journal	Impact Factor
New England Journal of Medicine	38.57
Journal of the American Medical Association	24.83
Lancet	21.71
Annals of Internal Medicine	13.11
Annual Review of Medicine	11.20
Archives of Internal Medicine	7.51
British Medical Journal	7.04
Canadian Medical Association Journal	5.94
American Journal of Medicine	4.18
Mayo Clinic Proceedings	3.75
Medicine	3.73
Annals of Medicine	3.62
Journal of Internal Medicine	3.59
American Journal of Preventive Medicine	3.19
Current Medical Research and Opinion	2.93
Journal of General Internal Medicine	2.82
QJM - An International Journal of Medicine	2.58
European Journal of Clinical Investigation	2.53
Preventive Medicine	2.33
Journal of Pain and Symptom Management	2.19

Medical Informatics Journal	Impact Factor
Journal of the American Medical Informatics Association	2.61
Statistical Methods in Medicine and Research	2.58
Medical Decision Making	1.68
IEEE Transactions on Information Technology in Biomedicine	1.58
Journal of Evaluation of Clinical Practice	1.56
Statistics in Medicine	1.39
Methods of Information in Medicine	1.34
International Journal of Medical Informatics	1.33
IEEE Engineering in Medicine and Biology Magazine	1.28
Artificial Intelligence in Medicine	1.12
Medical and Biological Engineering and Computing	1.07
International Journal of Technology Assessment	1.04
Journal of Biomedical Informatics	1.01
Biomedzinische Technik	0.83
CIN - Computers Informatics Nursing	0.82
Medical Informatics and the Internet in Medicine	0.72
Computer Merhods and Programs in Biomedicine	0.69
Journal of Cancer Education	0.51

Biotechnology Journal	Impact Factor
Nature Biotechnology	22.36
Nature Reviews Drug Discovery	19.58
Genome Research	10.38
Trends in Biotechnology	8.61
Current Opinion in Biotechnology	8.08
Pharmacogenetics	6.41
Bioinformatics	5.74
Stem Cells	5.50
Antisense and Nucleic Acid Drug Development	5.47
BMC Bioinformatics	5.42
Molecular Therapy	5.20
Gene Therapy	4.98
Human Gene Therapy	4.86
Molecular Plant-Microbe Interactions	4.05
Genomics	3.84
Applied Environmental Microbiology	3.81
Mutation Research - Fundamental and Molecular Mechanisms of Mutagenesis	3.73
Mutation Research - Reviews in Mutation Research	3.67
Cancer Gene Therapy	3.65
Metabolic Engineering	3.42

Information Systems Journal	Impact Factor
Annual Reivew of Information Science and Technology	4.29
ACM Transactions on Information Systems	4.10
VLDB Journal	4.00
MIS Quarterly	2.88
Journal of Chemical Information and Computer Science	2.81
Data Mining and Knowledge Discovery	2.80
IEEE Network	2.67
Journal of the American Medical Informatics Association	2.61
Journal of the ACM	2.41
IEEE Wireless Communications	2.19
Journal of the American Society for Information Science and Technology	2.09
IEEE Transactions on Information Theory	2.03
Distributed Parallel Databases	1.85
ACM Transactions of Database Systems	1.85
Information and Management	1.82
IBM Systems Journal	1.64
IEEE Transactions on Multimedia	1.63
IEEE Transactions on Information Technology in Biomedicine	1.58
Decision Support Systems	1.46
Wireless Networks	1.35
...
Information Retrieval (a journal I am Co-Editor-in-Chief of!)	1.23

2.3.5 Propagation

(3/31/04) Although not a section in the original book, interest in the notion of information propagation has been revived with the growth of the Internet and Web, which provides a vast new medium for information spread. The notion of the propagation of information can be traced back to Dawkins (1976), whose book laid out the ideas of memes, which are information patterns that are held in a person's memory but can be copied to another. The field that studies the replication and evolution of memes is called memetics. There are many Web sites devoted to memetics, e.g., (Heylighen et al., 1999).

Dawkins give examples of memes as "tunes, ideas, [and] catch-phrases," that propagate from "brain to brain." Memes have been likened to genes, but are more appropriately compared to T-phage viruses, which cannot replicate themselves but take over a cell's DNA to cause it to make millions of copies of itself (Heylighen et al., 1999). Memes can affect the mind like a parasite, causing an individual to change his or her behavior and/or pass the idea on to others. Memes are selected or, in genetic terms, have fitness by a variety of properties such as novelty, coherence, and self-reinforcement. If they do not have the capability to survive, then they may die out.

The Internet is a (relatively) new medium for the wide spread of memes. The frequent forwarding of emails as well as visiting of Web sites are common means for memes to propagate. One consequence of such easy spread of information is the propagation of misinformation, which are sometimes called "urban legends" (e.g., http://www.urbanlegends.com, http://www.snopes.com).

Heylighen, F., Joslyn, C. and Turchin, V. (1999). Principia Cybernetica Web. http://pespmc1.vub.ac.be/TOC.html. Accessed: October 20, 2003.
Dawkins, R. (1976). The Selfish Gene. New York. Oxford University Press.

2.4 A classification of textual health information

2.5 Production of health information

The editors of Annals of Internal Medicine recently reiterated that the scientific process must be reprducible and transparent. While noting that the day-to-day operations of science are largely unnoticed by the public, significant attention is paid when fraud is uncovered. To maintain the public's trust, they argue, evaluative processes must be in place to insure such trust is warranted.

Laine, C., Goodman, S., et al. (2007). Reproducible research: moving toward research the public can really trust. Annals of Internal Medicine, 146: 450-453.

2.5.1 The generation of scientific information

(4/3/05) Another way to think of the scientific literature is via its "life cycle." The figure below somewhat recapitulates what is described in the book, but also shows some of the additional wrinkles of the modern scientific discovery and publishing process.

Many also argue that there is much more to scientific data than publications, and that the sheer volulme of information is enormous. This is perhaps described most eloquently by Insel et al. (2003), who note the massive amounts of things like nucleotides in the genome and neurons in the brain. These authors also note that the scientific publication process is "slow and expensive," and advocate for more data sharing among researchers.

Of course, some knowledge is never obtained because it is "forbidden" to be studied (Kempner, 2005). Knowledge may be forbidden because it can only be obtained through unethical means, e.g., human experiments conducted by Nazis. But other research is prohibited by what Kempner et al. call "informal constraints." This may involve fear from results being attacked by political groups across the spectrum, from religious groups to animal rights activists. Clearly there must be some ethical constraints on the conduct of science, but not merely if they offend the political agenda of a particular group.

Related to forbidden knowledge is the focus of scientific literature on diseases and their treatments pertinent to developed countries. Raja and Singer (2004) note that little content in all of the "major" journals is relevant to developing countries, but that British journals do a better job than their American counterparts.

Kempner, J., Perlis, C., et al. (2005). Forbidden knowledge. Science, 307: 854.
Insel, T., Volkow, N., et al. (2003). Neuroscience networks: data-sharing in an information age. PLoS Biology, 1(1): E17. http://biology.plosjournals.org/plosonline/?request=get-document&doi=10.1371/journal.pbio.0000017.
Raja, A. and Singer, P. (2004). Transatlantic divide in publication of content relevant to developing countries. British Medical Journal, 329: 1429-1430.

(4/5/04) Another concern about the prodcution of biomedical literature is that the clinical trials carried out do not meet the needs of "decision makers," in particular, those who develop policy, practice guidelines, and so forth. Tunis et al. (2003) have called for more effort on pragmatic or practical clinical trials (PCTs). The characteristics of PCTs they deem most important include the selection of clinically relevant interventions for comparison, diverse populations of study participants, recruitment from heterogeneous practice settings, and data collection from a broad range of clinical outcomes. They lament that the major funders of clinical research, namely the National Institutes of Health and the medical products industry, do not focus on supporting these types of clinical trials.

Tunis, S., Stryer, D. and Clancy, C. (2003). Practical clinical trials - increasing the value of clinical research for decision making in clinical and health policy. Journal of the American Medical Association, 290: 1624-1632.

(4/12/06) The International Committee of Medical Journal Editors (ICMJE, www.icmje.org) has a document on its Web site that describes its missio and activities (Anonymous, 2006)

Anonymous (2006). Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication, International Committee of Medical Journal Editors. http://www.icmje.org/icmje.pdf.

2.5.2 Study of the peer review process

(4/4/07) Is peer review necessary in this era of widespread access to the Internet and World Wide Web? Some have suggested that posting submitted papers on-line and allowing public comment and update could substitute for it. To this end, the journal Nature carried out an experiment in 2006 of one approach to open peer review. For a four month period, they gave authors the option of having their submissions posted for public comment in addition to the usual peer-review process. Only 71 (5%) of the 1,369 authors submitting papers during the four months of the period agreed to take part. For these papers, comments were posted on only 38 (54%) of them. A total of 92 technical comments were posted (i.e., about 2.5 per paper commented upon). The number of comments varied widely by subject domain. The authors were asked about the utility of the comments, and most were found to be not helpful, though some editorial comments were found to be of value. The editors of Nature (2006) concluded that this approach would not be pursued further for now.

Anonymous (2006). Overview: Nature's peer review trial. Nature. http://www.nature.com/nature/peerreview/debate/nature05535.html.
Anonymous (2006). Peer review and fraud. Nature, 444: 971-972.

(4/9/06) Is the peer-review process "broken?" McCook (2006) notes that the growing numbers of submissions and the pressures for scientists to publish, especially in prestigious journals, to obtain promotion and/or continued grant funding, is taxing the system. This also leads to hyping the conclusions and downplaying the limitations of studies. McCook lists three specific complaints with the current peer-review system and quotes editors and others with suggestions for change:

Editors at commercial (e.g., Nature, Cell) are in general younger than those at society or non-profit journals and thus less experienced. It is unclear, however, that the age of an editor matters, as they may have more experience but also formed stronger opinions and networks of colleagues.
Journals with sister publications steer papers into them to increase their profiles. A deputy editor of JAMA is quoted as denying this.
Peer reviewers delay or otherwise sabotage reviews of competing scientists. The editor-in-chief is quoted as noting this is an "extreme exception."

The article discusses advantages for and against open peer review (i.e., the reviewer is identified to the authors), which is done at BMJ and BMC. Even though the evidence is unclear that the process works better, many editors believe it leads to more constructive reviews. Others caution that reviewers may lose obejctivity and fear repercussions, especially from senior leaders in the field. One suggestion for fixing peer review is the separation of improving their writing from judging their scientific merit (Kaplan, 2005).

A common practice in peer review is to ask authors for suggested reviewers. Does this make a difference? One analysis of 329 manuscripts from ten leading journals found that the quality of the reviews was judged similar but that author-suggested reviewers tended to make more favorable recommendations concerning publication (Schroter, 2006). The authors conclude that author-suggested reviewers can adequately review papers but their recommendations for publication should be more scrutinized.

Kaplan, D. (2005). How to fix peer review. The Scientist, 19(11): 10. http://www.the-scientist.com/article/display/15501/.
McCook, A. (2006). Is peer review broken? The Scientist, 20(2): 26-34. http://www.the-scientist.com/article/display/23061/.
Schroter, S., Tite, L., et al. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. Journal of the American Medical Association, 295: 314-317.

(4/9/06) Scientific information would not be generated at all were it not for research funding. The process to determine research is also done via peer review. The largest grantor of funding in the world is the US National Institutes of Health (NIH). Of its $28 billion annual budget, 80% supports research and training outside NIH. Each of the NIH's 20+ institutes allocates funds for research in a competitive process. (The major institute for biomedical informatics research is the NLM, which also receives substantial funding for its library operations.) The NIH receives about 80,000 grant proposals per year, recruiting more than 15,000 experts to review them. Proposals are usually group by subject and/or grant type, with a group of reviewers recruited to form a study section. The general process is to assign a priority score, from 100 (best) to 500 (worst), and then funding proposals by priority until funds are exhausted. The institutes have some leeway to adjust scores and prioritize funding. The director of the NIH's Center for Scientific Review recently provided an overview of the system and areas of concern (Scarpa, 2006). There is growing concern that the leveling off of NIH funding growth is difficult not only for individual scientists, but also the institutions who depend on them to get funding to cover portions of their salaries. There are also concerns at the long lead time required for funding decisions, on average about nine months. This not only delays research but also creates funding uncertainties for researchers and the institutions that employ them.

A related concern is that basic research is not being "translated" quickly enough into benefits for human health, with growing calls for more emphasis on "translational research" (Zerhouni, 2005). The peer review process of grant proposals has been less studied than peer review of journal articles. The most investigated aspect concerns funding rates of basic vs. clinical research. Two recent analyses show that applications for clinical research on average receive worse priority scores (Kotchen, 2004; Kotchen, 2006). Other findings include that smaller study sections result in worse priority scores for clinical although not basic science research. About 20-25% of submitted proposals are funded.

Kotchen, T., Lindquist, T., et al. (2004). NIH peer review of grant applications for clinical research. Journal of the American Medical Association, 291: 836-843.
Kotchen, T., Lindquist, T., et al. (2006). Outcomes of National Institutes of Health peer review of clinical grant applications. Journal of Investigative Medicine, 54: 13-19.
Scarpa, T. (2006). Research funding: peer review at NIH. Science, 311: 41.

(4/3/05) I agree with my colleague Tefko Saracevic, who has said (personal communication) that the peer review process determines more where an article is published than whether it is published. Another example of those who are already successful continuing to achieve success can be found in the book, The Jordan Rules (Smith, 1994), in which an analysis of calls by NBA referees found a tendency to give this (and probably other) superstars the benefit of the doubt in foul calls.

Smith, S. (1994). The Jordan Rules. New York, NY. Pocket Books.

(4/3/05) Another study of problems with the peer review process comes from the social work literature (Epstein, 2004). In this study, two "stimulus" articles, written with both a positive and negative interpretation, were submitted to 31 social work journals. The acceptance rates between the positive and negative versions were significant for one of the articles but not the other. The timeliness and quality of the peer reviews were considered inadequate in 73.5% of the reviews.

Epstein, W. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14: 450-458.

(4/4/04) The systematic review of the peer review process by Jefferson et al. (2002) described in the text showed that research has not demonstrated the benefits of this process. Twenty-one studies of the process were found and led to a variety of conclusions (number supporting each conclusion in parentheses):

Concealing identities of peer reivewers or authors does not appear to affect quality of reviews (9)
Checklists and other attempts at standardizing the process do not appear to help (2)
Training of referees does not improve the quality of reviews (2)
Electronic media do not improve quality (2)
Peer review does not detect bias against unconventional drugs (1)
The process may improve readability and general quality of papers (2)

Jefferson, T., Wager, E. and Davidoff, F. (2002). Measuring the quality of editorial peer review. Journal of the American Medical Association, 287: 2786-2790.

2.5.3 Primary literature and its limitations

(4/10/06) There has been a great deal of concern in recent years with the economics and politics of randdomized controlled trials (RCTs). From the perspective of evidence-based medicine (EBM), RCTs are the undisputed king of evidence. There is growing concern, however, that RCTs too have limitations, perhaps the most significant of which are the consequence of their substantial economic value, particularly to pharmaceutical companies.

The book already describes some of the limitations of RCTs, such as that the quality of study is inversely related to magnitude of treatment effect (Moher, 1998) and well-designed observational studies may be just as good (Benson, 2000). More recently, it has been found that lower-quality studies are more likely to be later "overturned" (Ioannidis, 2005). This is probably due to their relatively small sample sizes that prevent complete knowledge of all effects, especially relatively uncommon adverse ones. A related problems occurs with studies that are stopped early because a strong and statistically significant treatment effect has occured (Montori, 2005). These studies are usually published in the major five medical journals and funded by industry. However, despite this happening more commonly, they often fail to fully document their reasons for stopping early.

Another problem with articles about RCTs is inadequate reporting of adverse events. Chan et al. (2004) assessed 102 clinical trials and their clinical outcome measures that were approved by ethics committees in Denmark during 1994-1995. They found that 50% of the efficacy outcomes and 65% of the harm outcomes were incompletely reported. About 62% of the trials had at least one clinical outcome that had been changed, introduced, or omitted from the original study protocol. A survey of trial authors denied the existence of unreported outcomes despite their existence identified by Chan et al.. More recently, Fromme et al. (2004) have shown that chemotherapy trials do not document all the adverse effects that occur. It is clearly important that post-marketing surveillance is still essential (Fontanarosa, 2004). Another limitation of RCTs is that some clinical interventions require training or skill that is not always taken into account. One way to address this that has been proposed in "expertise'-based trials (Devereaux, 2005).

A related problem in the publication of RCTs is selective reporting. The concerns about pharmaceutical company advertising are well-known, but perhaps a more serious (because it is less evident) problem is the manipulation of studies and results reported (Smith, 2005). Threre are a variety of techniques that can be used:

Conduct a trial against a drug known to be inferior
Conduct a trial against a competitor at too low (to get better result) or too high (to have less toxicity) of a dose
Conduct trials with samples too small to show a difference from competitor
Use multiple endpoints and select those that show best benefit
Do multicenter trials but only report results from centers that are favorable
Conduct subgroup analysis and only report those that are favorable
Report results most likely to impress, i.e., report relative rather than absolute risk reduction

There are a number of concerns with RCTs funded by the pharmaceutical industry. About 66-75% of all trials published in major medical journals are funded by industry (Egger, 2001). In a systematic review of studies comparing research sponsored by the pharmaceutical industry with that sponsored by others, Lexchin et al. (2003) analyzed 30 studies and found that the latter was more likely to have a positive outcome and less likely to be published. Their analysis did not, however, find these studies were of poorer quality. The better outcomes were explained by inappropriate comparator products and publication bias. Perlis (2005) has documented similar problems in the psychiatry literature. The former editor of the British Medical Journal, Smith (2005) has noted that publication of studies in journals is lucrative. They look more “professional” than advertisements and the paper reprints are a large source of revenue for journals, up to 70% profit margin. This has led him to call the medical journals and pharmaceutical companies "uneasy bedfellows" (Smith, 2003). Other journals have lamented the use of corporate "ghost writing" to hone the message in papers that report RCTs (Tierney, 2005; Laine, 2005).

A case study of RCT reporting gone awry comes from the COX-2 inhibitor drugs. These are a class of non-steroidal anti-inflammatory drugs (NSAIDs) use to control pain, especially in patients with chronic and/or inflammatory conditions, such as the different types of arthritis. NSAIDS are commonly used and have serious gastrointestinal (GI) adverse effects, resulting in 103,000 hospitalizations and 16,500 deaths per year (Wolfe, 1999). COX-2 inhibitors were hypothesized to selectively provide anti-inflammatory efficacy of NSAIDS without GI side effects. Two large RCTs were found to have selectively reported their results subsequent to their publication. Combined with the adverse results from another trial, they were removed from the market (Malhorta, 2004). A review of these RCTs is informative.

The CLASS trial assessed celecoxib (Celebrex) versus ibuprofen and diclofenac (two older NSAIDs) (Silverstein, 2000). The paper about the study reported fewer peptic ulcers in celecoxib at 6 months. However, the paper did not report additional information reported to FDA (and published on FDA Web site) that by one year, differences were no longer present (Hrachovec, 2001; Wright, 2001). The paper authors decided to not report second 6 months of data because they believed it was invalid (Silverstein, 2001). Carrying out the analysis of the study as originally planned for trial would not have led to conclusions in original report (Jüni, 2002). Jüni et al. also noted that the paper had already been cited 169 times while these issues came to light and that the company ordered 30,000 reprints from the publisher. Clearly the stakes of publications in journals can be high.

The RIGOR trial compared rofecoxib (Vioxx) versus naproxen (Bombardier, 2000). The paper reporting the study described fewer GI side effects but more overall serious adverse events, which were downplayed in paper. All of the authors of the paper had ties to or were employed by Merck (the manufacturer of Vioxx). Furthermore, it later came to light that three patients suffering MI were not included in paper data despite authors knowing about them (Curfman, 2005). This would have increased the reported relative risk of rofecoxib and made the difference statistically significant in patients for whom aspirin not indicated (i.e., those without heart disease).

The trial that sealed the fate of the COX-2 inhibitors was the APPROVe trial which assessed whether they reduce recurrent colonic polyps (Breslaier, 2005). Laboratory evidence had suggested that COX-2 inhibitors may reduce the recurrence of colonic polyps. These researchers undertook an RCT to answer this question, but stopped the study prematurely due to excess thrombotic events that appeared after 18 months.

The outrage over the changes in the CLASS and RIGOR trial after their inception increased the long-standing calls for pre-trial registration. Long advocated by many EBM advocates to protect integrity of RCT conduct and reporting, registration requires those conducting RCTs to register them with details of hypotheses, methods, etc. (Dickersin, 2003). Therefore, any changes that are made must be documented and scrutinized. After the COX-2 and related (e.g., SSRI inhibitors and suicide, Lancet, 2004) debacles, the ICMJE adopted a policy of requiring it at inception of study (Deangelis, 2005). Now, RCTs must be registered in ClinicalTrials.gov (Zarin, 2005) or other comparable databases (Haug, 2005). This has in substantial registration of RCTs. In addition, the new Ottawa Statement has been adopted that reflects principles of data to be entered (Krleza-Jeric, 2005).

A number of other solutions have been proposed to the influence of those who stand to benefit from the publication of pharmaceutical RCTs. One extreme proposal is that journals should not publish RCTs (Smith, 2005). Instead, advocates Smith, there should be more public funding of RCTs, especially large head-to-head ones, and RCT protocols and results should be made available on Web sites with the role of journals being instead to critically critique them. If RCTs are to be published elsewhere, where would that be? A number of clinical trials results databases have been developed, none of which has comprehensive coverage and for which concerns about lack of peer review have been expressed (Fisher, 2006). Some have suggested increased use of the US Food and Drug Administration (FDA) Dockets Management system (http://www.fda.gov/ohrms/dockets/default.htm), which contains information required of pharmaceutical companies for FDA approval of their drugs. Many researchers find it a source of clinical trials data beyond that which is reported in articles. However, the site is extremely user-unfriendy, which is thought to be deliberate by some. Another possibility would be the use of the new drug application (NDA) database of the US Food and Drug Administration (Turner, 2004). A different proposal is to develop the Global Trial Bank (Sim, 2005), which is promoted by the American Medical Informatics Association (http://www.amia.org/gtb/). The Global Trial Bank is based on capturing all the elements believed necessary to allow others to analyze all data of trial (Sim, 2004). Unlike the CONSORT statement (Moher, 2001), it reports all data.

Another concern is the conflict of interest between those who perform studies for the industry as well as those who develop clinical practice guidelines. Henry et al. (2005) found that Austalian clinicians who perform research for pharmaceutical companies are more likely to receive gifts and/or obtain support for travel to international conferences. Likewise, Taylor et al. (2005) found that 35% of 685 authors of guidelines reported a conflict of interest and that half of all guidelines published do not even have any declaration of conflict of interest. Related to this, pharamaceutical sales representatives are trained to downplay the risks of drugs when promoting their use (Waxman, 2005). Researchers may also be bound by gag clauses that limit their ability to publish or report certain adverse events (Steinbrook, 2005). All of these findings have led for leaders in medicine to call for more stringent policies limiting industry support, especially in academic medical centers (Brennan, 2006).

Bombardier, C., Laine, L., et al. (2000). Comparison of upper gatrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. New England Journal of Medicine, 343: 1520-1528.
Brennan, T., Rothman, D., et al. (2006). Health industry practices that create conflicts of interest: a policy proposal for academic medical centers. Journal of the American Medical Association, 295: 429-433.
Bresalier, R., Sandler, R., et al. (2005). Cardiovascular events associated with rofecoxib in a colorectal adenoma chemoprevention trial. New England Journal of Medicine, 352: 1092-1102.
Chan, A., Hrobjartsson, A., et al. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association, 291: 2457-2465.
Curfman, G., Morrissey, S., et al. (2005). Expression of concern: Bombardier et al., "Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis". New England Journal of Medicine, 353: 2318-2319.
DeAngelis, C., Drazen, J., et al. (2005). Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. Journal of the American Medical Association, 293: 2927-2929.
Devereaux, P., Bhandari, M., et al. (2005). Need for expertise based randomised controlled trials. British Medical Journal, 330: 88.
Dickersin, K. and Rennie, D. (2003). Registering clinical trials. Journal of the American Medical Association, 290: 516-523.
Egger, M., Bartlett, C., et al. (2001). Are randomised controlled trials in the BMJ different? British Medical Journal, 323: 1253-1254.
Fisher, C. (2006). Public health. Clinical trials results databases: unanswered questions. Science, 311: 180-181.
Fontanarosa, P., Rennie, D., et al. (2004). Postmarketing surveillance - lack of vigilance, lack of trust. Journal of the American Medical Association, 292: 2647-2650.
Fromme, E., Eilers, K., et al. (2004). How accurate is clinician reporting of chemotherapy adverse effects? A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30. Journal of Clinical Oncology, 22: 3485-3490.
Haug, C., Gotzsche, P., et al. (2005). Registries and registration of clinical trials. New England Journal of Medicine, 353: 2811-2812.
Henry, D., Doran, E., et al. (2005). Ties that bind: multiple relationships between clinical researchers and the pharmaceutical industry. Archives of Internal Medicine, 165: 2493-2496.
Hrachovec, J. and Mora, M. (2001). Reporting of 6-month vs 12-month data in a clinical trial of celecoxib. Journal of the American Medical Association, 286: 2398.
Ioannidis, J. (2005). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294: 218-228.
Jüni, P., Rutjes, A., et al. (2002). Are selective COX 2 inhibitors superior to traditional non steroidal anti-inflammatory drugs? British Medical Journal, 324: 1287-1288.
Krleza-Jeric, K., Chan, A., et al. (2005). Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). British Medical Journal, 330: 956-959.
Laine, C. and Mulrow, C. (2005). Exorcising ghosts and unwelcome guests. Annals of Internal Medicine, 143: 611-612.
Lexchin, J., Bero, L., et al. (2003). Pharamceutical industry sponsorship and research outcome and quality: systematic review. British Medical Journal, 326: 1167-1170.
Malhotra, S., Shafiq, N., et al. (2004). COX-2 inhibitors: a CLASS act or Just VIGORously promoted. MedGenMed, 6(1): 6. http://www.medscape.com/viewarticle/470342.
Montori, V., Devereaux, P., et al. (2005). Randomized trials stopped early for benefit: a systematic review. Journal of the American Medical Association, 294: 2203-2209.
Perlis, R., Perlis, C., et al. (2005). Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. American Journal of Psychiatry, 162: 1957-1960.
Silverstein, F., Faich, G., et al. (2000). Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study: a randomized controlled trial. Journal of the American Medical Association, 284: 1247-1255.
Silverstein, F., Simon, L., et al. (2001). Reporting of 6-month vs 12-month data in a clinical trial of celecoxib - In reply. Journal of the American Medical Association, 286: 2399-2400.
Sim, I., Olasov, B., et al. (2004). An ontology of randomized controlled trials for evidence-based practice: content specification and evaluation using the competency decomposition method. Journal of Biomedical Informatics, 37: 108-119.
Sim, I. and Detmer, D. (2005). Beyond trial registration: a global trial bank for clinical trial reporting. PLoS Medicine, 2(11): e65. http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020365.
Smith, R. (2003). Medical journals and pharmaceutical companies: uneasy bedfellows. British Medical Journal, 326: 1202-1205.
Smith, R. (2005). Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Medicine, 2(5): e138. http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020138.
Steinbrook, R. (2005). Gag clauses in clinical-trial agreements. New England Journal of Medicine, 352: 2160-2162.
Taylor, R. and Giles, J. (2005). Cash interests taint drug advice. Nature, 437: 1070-1071.
Tierney, W. and Gerrity, M. (2005). Scientific discourse, corporate ghostwriting, journal policy, and public trust. Journal of General Internal Medicine, 20: 550-551.
Turner, E. (2004). A taxpayer-funded clinical trials registry and results database. PLoS Medicine, 1: 180-182.
Waxman, H. (2005). The lessons of Vioxx - drug safety and sales. New England Journal of Medicine, 352: 2576-2578.
Wolfe, M., Lichtenstein, D., et al. (1999). Gastrointestinal toxicity of nonsteroidal antiinflammatory drugs. New England Journal of Medicine, 340: 1888-1899.
Wright, J., Perry, T., et al. (2001). Reporting of 6-month vs 12-month data in a clinical trial of celecoxib. Journal of the American Medical Association, 286: 2398-2399.
Zarin, D., Tse, T., et al. (2005). Trial Registration at ClinicalTrials.gov between May and October 2005. New England Journal of Medicine, 353: 2779-2787.

(4/4/07) Another persistent problem is research fraud. Unfortunately, the extent of the problem is not known, since science operates, for the most part, on the "honor system," where the methods and results of researchers are presumed to be honest until proven otherwise. One recent prominent case of fraud concerns the stem cell research of Woo Suk Hwang of South Korea. Hwang was viewed as a national here after having made reported breakthroughs in the use of stem cells, which show promise in many human diseases but are controversial due to the only method for obtaining them being from the umbilical cords of aborted fetuses. Two papers published in Science were viewed as particularly seminal breakthroughs (Hwang, 2004; Hwang, 2005). However, a Korean investigative news show was tipped off by some of Hwang's collaborators (Chong, 2006). Ultimately, all of his work was declared to be frauduent, and Science retracted the two seminal papers (Kennedy, 2006). A retrospective analysis found a number of problems in the Science papers that were not detected by the peer-review process (Couzin, 2006).

As noted in the book but again exemplified by the Hwang case, considerable challenges remain in the literature and its bibliographic databases when fraud is uncovered (Couzin, 2006; Unger, 2006). The first concerns the articles themselves. Paper copies remain on the shelves of libraries and photocopied in the files of researchers and others, with considerable doubt over how to indicate their fraudulence. Electronic publication may offer the hope of removing the articles or prominently announcing their retractions. In the case of the latter, however, the retraction still may be overlooked. Furthermore, the practice of posting PDFs or other versions of articles elsewhere on the Web may also cause the retraction to be missed. Electronic bibliographic databases such as MEDLINE can also announce the retraction. However, two problems remain. First, what about articles that cite the retracted works? While probably still valid themselves, the rationale for the research may be undermined. Second, what about the other papers of the author(s) that were not retracted? Do they represent fraud, or can their results be trusted?

Other documented cases of fraud abound (Couzin, 2006; Unger, 2006). The NIH releases notices of scientific misconduct when fraud by researchers funded by its grants is uncovered, e.g., (Anonymous, 2005a; Anonymous, 2005b; Anonymous, 2006). Science has raised the question of scientific misconduct in other areas, including "bubble fusion" (Service, 2006a) and chemical catalyzation (Service, 2006b). Annals of Internal Medicine reports a number of lessons learned, including the unwillingness of journals and academic institutions to take the necessary steps to insure "cleansing" of the literature from the case of fraudulent research by Eric Poehlman (Sox, 2006)

One case where fraud has not been documented but serious concerns remain about the research is in a study that assessed the value of distant prayer (from the US and Australia) to facilitate in-vitro fertilization in Korea (Cha, 2001). One of the authors was a department chairman at Columbia University, and has since removed his name from the paper. However, the paper is still indexed in MEDLINE and on the Journal of Reproductive Medicine Web site without any hints about these concerns. Flamm (2002, 2004a, 2004b) has documented signfificant problems with this research and the way concerns about it have been handled. It should be noted that an RCT of cardiac bypass patients receiving prayer found that it offered no benefit (Benson, 2006).

Anonymous (2005a). Findings of Scientific Misconduct. Bethesda, MD, National Institutes of Health. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-05-049.html.
Anonymous (2005b). Findings of Scientific Misconduct. Bethesda, MD, National Institutes of Health. http://grants1.nih.gov/grants/guide/notice-files/NOT-OD-05-064.html.
Anonymous (2006). Findings of Scientific Misconduct. Bethesda, MD, National Institutes of Health. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-06-059.html.
Benson, H., Dusek, J., et al. (2006). Study of the therapeutic effects of intercessory prayer (STEP) in cardiac bypass patients: a multicenter randomized trial of uncertainty and certainty of receiving intercessory prayer. American Heart Journal, 151: 934-942.
Cha, K. and Wirth, D. (2001). Does prayer influence the success of in vitro fertilization-embryo transfer? Report of a masked, randomized trial. Journal of Reproductive Medicine, 46: 781-787. http://www.reproductivemedicine.com/Features/2001/2001Sep.htm.
Chong, S. and Normile, D. (2006). How young Korean researchers helped unearth a scandal. Science, 311: 22-25.
Couzin, J. (2006). And how the problems eluded peer reviewers and editors. Science, 311: 23-24.
Couzin, J. and Unger, K. (2006). Scientific misconduct - cleaning up the paper trail. Science, 312: 38-43.
Flamm, B. (2002). Review of Cha, KY, Wirth, DP, Lobo, RA. Does prayer influence the success of in vitro fertilization-embryo transfer? The Scientific Review of Alternative Medicine, 6: 47-50.
Flamm, B. (2004a). The Columbia University 'miracle' study: flawed and fraud. Skeptical Inquirer. http://www.csicop.org/si/2004-09/miracle-study.html.
Flamm, B. (2004b). The Columbia University 'Miracle' Story. Round Rock, TX, Improving Medical Statistics and the Interpretation of Medical Studies. http://www.improvingmedicalstatistics.com/Columbia%20Miracle%20Study1.htm.
Hwang, W., Ryu, Y., et al. (2004). Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst. Science, 303: 1669-1674.
Hwang, W., Roh, S., et al. (2005). Patient-specific embryonic stem cells derived from human SCNT blastocysts. Science, 308: 1777-1783.
Kennedy, D. (2006). Editorial retraction. Science, 311: 335.
Service, R. (2006a). Researchers raise new doubts about 'bubble fusion' reports. Science, 311: 1532-1533.
Service, R. (2006b). Columbia lab retracts key catalysis papers. Science, 311: 1533.
Sox, H. and Rennie, D. (2006). Research misconduct, retraction, and cleansing the medical literature: lessons from the Poehlman case. Annals of Internal Medicine, 144: 609-613.
Unger, K. and Couzin, J. (2006). Scientific misconduct - even retracted papers endure. Science, 312: 40-41.

(4/3/05) Which journals publish the most high-quality (i.e., evidence-based) studies? McKibbon et al. (2004) looked at publications that provide summaries of "clinically important" articles (e.g, ACP Journal Club, Evidence-Based Medicine), finding that in a given area, a small number of journals publish the lion's share of high-quality clinical studies. They assessed 60,352 articles in 170 journals and found the following results by field:

In internal medicine (ACP Journal Club), four titles provide 56.5% of articles, while 27 supply the rest.
In general/family medicine (Evidence-Based Medicine), five titles provide 50.7% of articles, while 40 supply the rest.
In nursing (Evidence-Based Nursing), seven titles provide 51.0% of the articles, while 34 supply the rest.
In mental health (Evidence-Based Mental Health), nine titles provide 53.2% of the articles, while 34 supply the rest.

For the medical (but not the nursing) fields, there was a correlation between number of clinically important articles and the journal's IF.

McKibbon, K., Wilczynski, N., et al. (2004). What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals. BMC Medicine, 2: 33. http://www.biomedcentral.com/1741-7015/2/33.

(4/7/05) A whole host of other limitations continue to impact the biomedical literature. These problems do not necessarily invalidate the literature or imply that the process is fatally flawed. But they do remind us that we must be aware of the limitations of the scientific process and strive to minimize them.

One problem that was described in the book and continues now is statistical reporting. An analysis by Garcia-Berthou and Alcaraz (2004) found that one or more incongruencies occured with statistical reporting in the pretigious journals Nature and British Medical Journal in 38% and 25% of papers respectively. In 12% of these instances, the significance levels (P value) could be incorrect by an order of magnitude or more. Most errors were presumed to be due to rounding, transcription, or type-setting problems.

Another continuing problem is errors in references. Aronsky et al. (2005) assessed this problem in the biomedical informatics literature and found problems similar to those described in earlier studies in other scientific publications. These authors assessed the five biomedical informatics journals with the highest IFs for each journal's first issue of 2004. They found 311 errors in 225 of the 656 references (34.3%) in 37 articles. The percentage of articles with errors varied by journal, from 22.1% for Journal of the American Medical Informatics Association to 40.7% for International Journal of Medical Informatics. The most common element with an error was the author name (31%), followed by the title (17%), page (7.4%), and year (3.5%).

A new wrinkle to the problem of inadequate citations is Web references provided in scientific papers that are inaccessible or incorrect. Crichlow et al. (2004) assessed URLs in the references of all original research papers in five major medical journals that were published in January, 2004. In 91 articles analyzed, there were 68 URLs in the references, 8.6% of which were inaccessible. These authors noted that de Lacey et al. (1985) had found a similar 8% overall rate of errors in citations in the paper-based journal literature in 1985.

Despite the improvement in abstracts with the introduction of structured abstracts in the 1980s, they typically do not mention limitations of studies. In 2004, the Annals of Internal Medicine (2004) introducted a new section to their structured abstract, Limitations.

Anonymous (2004). Addressing the limitations of structured abstracts. Annals of Internal Medicine, 140: 480-481.
Crichlow, R., Winbush, N., et al. (2004). The accessibility and accuracy of Web references in five major medical journals. Journal of the American Medical Association, 292: 2723-2724.
Aronsky, D., Ransom, J., et al. (2005). Accuracy of reference in five biomedical informatics journals. Journal of the American Medical Informatics Association, 12: 225-228.
de Lacey, G., Record, C., et al. (1985). How accurate are quotations and references in medical journals? British Medical Journal, 291: 884-886.
Garcia-Berthou, E. and Alcaraz, C. (2004). Incongruence between test statistics and P values. BMC Medical Research, 4: 13. http://www.biomedcentral.com/1471-2288/4/13.

(4/3/04) A further systematic review of the fate of biomedical meeting abstracts verified that only about 46% of abstracts presented at such meetings achieve publication of the full paper in a journal (von Elm et al., 2003). These authors also reviewed the fate of abstracts originally rejected, finding 27% were eventually published as full papers. Studies in basic science and those having a positive outcome were more likely to eventually be published as papers. Abstracts were more likely to be published if they were presented orally, at a small meeting, or a US meeting.

von Elm, E., Costanza, M., Walder, B. and Tramer, M. (2003). More insight into the fate of biomedical meeting abstracts: a systematic review. BMC Medical Research Methodology, 3: 12. http://www.biomedcentral.com/1471-2288/3/12.

(4/12/06) There are additional concerns about compromised validity of information in journals short of outright fraud. One oft-cited culprit is the pharamceutical industry, both through its influence on the content as well as advertising in journals. In an editoral, Fletcher (2003) notes that advertisements are a major source of revenue for journals and provide resources to support the journal or the organization (often a professional society) that publishes it. However, he notes that while physicians claim not to have their practices influenced by advertisements, the advertisers would unlikely spend thousands of dollars per physician per year that they do if they had no effect.

Advertisements themselves, which readers encounter alongside the scientific papers in journals, can be misleading in their content. Wilkes et al. (1992) found that 44% of advertising would lead to improper prescribing if the physician had no other information. They also noted that 92% of advertisements included at least one area that did not comply with regulations of the FDA. Villanueva et al. (2003) looked at all advertisements for blood pressure-lowering and lipid-lowering medications in six Spanish medical journals during 1997. In a sample of references cited in the advertisements, they found that while 18% of the references could not be retrieved. In addition, 44% of the claims made were not completely supported by the reference, usually due to the drug being recommended in a patient group other than which it was studied.

Also problematic in advertisements may be the graphics. Cooper et al. (2003) analyzed all ads in ten US medical journals in 1999. Half of the ad area consisted of nonscientific figures and images. About 1.6% of the area contained scientific graphs. Over a third had some numerical distortion that led to overestimation or underestimation of the quantity being graphed, which is specifically prohibited by FDA regulations.

Another area of concern described in the text is conflict of interest. A recent episode led to the partial retraction of a paper by Lancet, when it was discovered that the primary author did not disclose funded by a group of lawyers representing alleged victims autism due to the measles, mumps, and rubella (MMR) vaccine (Horton, 2004). This demonstrates that conflict of interests are not necessarily limited to those who stand to gain from sale of products.

Fletcher, R. (2003). Adverts in medical journals: caveat lector. Lancet, 361: 10-11.
Wilkes, M., Doblin, B. and Shapiro, M. (1992). Pharmaceutical advertisements in leading medical journals: experts' assessments. Annals of Internal Medicine, 116: 912-919.
Villanueva, P., Piero, S., Librero, J. and Pereiro, I. (2003). Accuracy of pharmaceutical advertisements in medical journals. Lancet, 361: 27-32.
Cooper, R., Schriger, D., Wallace, R., Mikulich, V. and Wilkes, M. (2003). The quantity and quality of scientific graphs in pharmaceutical advertisements. Journal of General Internal Medicine, 18: 294-297.
Horton, R. (2004). A statement by the editors of The Lancet. Lancet, 363: 820-821.

2.5.4 Meta-analysis and its limitations

(4/11/06) This section should really be titled, Systematic reviews and their limitations, with a focus on systematic reviews that may or may not be able to use the technique of meta-analysis. A key point about systematic reviews is that they represent a systematic attempt to bring together all the knowledge in a given area. If it is appropriate to pool results of experiments quantitatively, this can be done via meta-analysis. But even if meta-analysis is not done, just the process of a systematic review can generate new knowledge by giving insights to the overall picture of studies in an area.

What are the characteristics of systematic reviews published in the medical literature? Montori et al. (2003) assessed 170 well-known clinical journals for the year 2000, counting 60,330 articles. Of these articles, 26,694 were original research reports and 3,193 were review articles. Of the review articles, 768 (24%) were systematic reviews, defined as articles that clearly stated a clinical topic, how the evidence was retrieved, what sources the evidence was retrieved from, and what the inclusion and exclusion criteria were. The majority of systematic reviews were about therapy (63%), followed by causation and safety (29%), diagnosis (4.4%), and prognosis (2.1%). About 80% of all the systematic reviews were published in 11% of the journals. The IF of these journals was weakly but significantly assocation with the publication of systematic reviews. Systematic reviews were more likely to be cited by other papers in these journals than narrative reviews.

One aspect that characterizes systematic reviews is their reporting of explicit search strategies and assessment of their effectiveness. Patrick et al. (2004) found that while the majority of meta-analyses (71%) reported a search strategy, only a small number (6.7%) reported evidence of the strategy's effectiveness.

Several references describe the methods for producing systematic reviews in great detail: the Cochrane Reviewers' Handbook (Anonymous, 2005) and the book by Glasziou et al. (2001).

The Annals of Internal Medicine recently published a supplement describing the challenges of summarizing information (i.e., producing systematic reviews or evidence reports), with a focus on work done by the Evidence-Based Practice Center (EPC) program of the Agency for Healthcare Research and Quality (AHRQ) (Helfand, 2005). The supplement includes an overview of the EPC program and articles about challenges in various areas, such as efficacy of drugs (Santauigda, 2005), use of nonrandomized studies (Norris, 2005), and dissemination of reports (Matchar, 2005).

Anonymous (2005). Cochrane Reviewers' Handbook. Cochrane Collaboration. http://www.cochrane.org/resources/handbook. Accessed: March 27, 2005.
Glasziou, P., Irwig, L., et al. (2001). Systematic Reviews in Health Care: A Practical Guide. Cambridge, UK. Cambridge University Press.
Helfand, M., Morton, S., et al. (2005). Challenges of Summarizing Better Information for Better Health: The Evidence-based Practice Center Experience. Annals of Internal Medicine, 142(12 - Part 2).
Matchar, D., Westermann-Clark, E., et al. (2005). Dissemination of Evidence-based Practice Center reports. Annals of Internal Medicine, 142: 1120-1125.
Montori, V., Wilczynski, N., et al. (2004). Systematic reviews: a cross-sectional study of location and citation counts. BMC Medicine, 1: 2. http://www.biomedcentral.com/1741-7015/1/2.
Norris, S. and Atkins, D. (2005). Challenges in using nonrandomized studies in systematic reviews of treatment interventions. Annals of Internal Medicine, 142: 1112-1119.
Patrick, T., Demiris, G., et al. (2004). Evidence-based retrieval in evidence-based medicine. Journal of the Medical Library Association, 92: 196-199.
Santaguida, P., Helfand, M., et al. (2005). Challenges in systematic reviews that evaluate drug efficacy or effectiveness. Annals of Internal Medicine, 142: 1066-1072.

(4/11/06) One challenge for meta-analysis is duplicate publication (which is actually a problem beyond meta-analysis), since it results in single individuals being counted more than once, potentially effecting the results of the analysis. How prevalent is the problem of duplicate publication? von Elm et al. (2004) analyzed 141 systematic reviews published in anesthesiology and available on the Internet. Of these reviews, the authors of 56 acknowledged identification of duplicate articles (excluding abstracts, letters, and book chapters), leading them to identify 103 duplicates of 78 articles (60 were published twice and the remainder more than twice). The duplicates were not mere reproductions, but fell into more complex (one might say covert) patterns (number of article pairs in parentheses):

Study samples identical

Outcomes identical - one report (21) or more than one report (16)
Outcomes different (24)

Study samples different

Outcomes identical - increasing sample (11) or decreasing sample (11)
Outcomes different (20)

All but 5.3% of the papers referenced the earlier duplicates. Two-thirds differed in authorship partially or completely. The annual citation rate was about equal for each in the pair. The median appearance of the duplicate was at about one year.

Another analysis looked at studies assessing the effect of the drug odansetron in postoperative emesis (Tramer, 1997). They identified a total of 84 RCTs that included 11,980 patients receiving odanseteron published from 1991-1996. Data from nine RCTs were published in 14 further reports (17% of trials) represeting 3,335 (28%) patients. The overall efficacy of the drug in these trials was positive, but studies that weer duplicated tended to show more positive results, with a 23% overestimation of efficacy of the drug.

Tramer, M., Reynolds, D., et al. (1997). Impact of covert duplicate publication on meta-analysis: a case study. British Medical Journal, 315: 635-640.
von Elm, E., Poglia, G., Walder, B. and Tramer, M. (2004). Different patterns of duplciate publication: an analysis of articles used in systematic reviews. Journal of the American Medical Association, 291: 974-980.

(4/4/04) Another analysis of the grey (unpublished) literature was at odds with the analysis of McAuley et al. (2002) cited in the text. Hopewell et al. (2003) found that published trials demonstrated an overall larger treatment effect. This analysis also found that published trials were were likely to be larger and of higher methodologic quality. Grey literature most commonly consisted of either abstracts (49%) or unpublished data (33%).

Hopewell, S., McDonald, S., Clarke, M. and Egger, M. (2003). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Library. http://www.cochrane.org/cochrane/mrabstr/mr000010.htm.

(4/4/04) Despite the prevalence of IR systems, handsearching of the literature is still required to identify all trials to include in meta-analyses. Thirty-four assessments in a variety of topical areas have demonstrated that handsearching yields 92-100% of all reports of RCTs, whereas searching of MEDLINE and other databases reveals only 49-67% (Hopewell et al., 2003).

Hopewell, S., Clarke, M., Lefebvre, C. and Scherer, R. (2003). Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Library. http://www.cochrane.org/cochrane/mrabstr/mr000001.htm.

2.5.5 Secondary literature and its limitations

(4/4/04) The text notes that new research findings are often slow to appear in textbooks. However, another problem in textbooks concerns the promulgation of information not very well investigated in the first place. One area where this has been found to be problematic concerns findings on physical examination in diseases. Richardson and Wilson (2002) have noted, for example, that frequencies of findings in diseases are often not described in popular internal medicine textbooks.

In this era of continually increasing costs, there is an oft-stated desire for physicians to rely more on the physical exam and less on expensive testing. Medical textbooks are particularly noteworthy for their description of eponymous physical findings, often which are used by elderly attending physicians to demonstrate their knowledge. Most of these findings, however, have been less subject to the modern scrutiny of diagnostic test evaluation. Babu et al. (2003) looked at 12 eponymous signs of aortic regurgitation, the condition where the aortic valve does not close completely and allows blood to "leak" back into the heart. This can lead to congestive heart failure and other complications. While these signs are described in major textbooks, the actual evidence supporting them is for the most part minimal. The authors call for physical findings to be evaluated using the more modern techniques of evidence-based medicine (see Section 2.8).

Richardson, W. and Wilson, M. (2002). Textbook descriptions of disease - where's the beef? ACP Journal Club, 137: A11-A12.
Babu, A., Kymes, S. and Carpenter-Fryer, S. (2003). Eponyms and the diagnosis of aortic regurgitation: what says the evidence? Annals of Internal Medicine, 138: 736-742.

2.6 Electronic publishing

(4/3/05) Electronic publishing of scientific journals has continued to evolve since publication of the book. Virtually all biomedical journals are now available in electronic form, and academic medical center libraries subscribe to hundreds if not thousands. The biggest challenge now are the non-technical issues, particularly economic ones. This has motivated the advocacy of "open access" publishing as described below.

Another major shake-up to the publishing world is Google Scholar (scholar.google.com) (Henderson, 2005). Google Scholar contains scientific publications it has found from its crawling the Web for content in its regular system. It also has established linkages across publications. Retrieved articles are ranked by the number of citations to the articles, which is a variant of the Google PageRank algorithm (see Chapter 5). Both Google Scholar and the plain Google search engine have impacted searching for electronic publications. Many MEDLINE references as well as full-text articles are detected and indexed by the Google crawler. Analysis of journals published electronically by Highwire Press (http://www.highwire.org/) (Steinbrook, 2006) as well as just the BMJ (Giustini, 2005) have found that over half of all accesses of their full-text articles come from links from Google or Google Scholar. Not only are people using Google to find articles, but there is at least one report of symptoms being entered into it to make a diagnosis (Greenwald, 2005).

Another Google effort related to electronic publishing in biomedicine is the Google Book Search initiative (previously called Google Print). Google has entered into an agreement with five large and prestigous universities to digitize their collections (Markoff and Wyatt, 2004). Non-copyrighted works will be completely available, while only excerpts of copyrighted publications will be accessible. The process of capturing documents with high-reolution cameras will be very labor-intensive. Google's effort is not the only large-scale one to digitize collections; the Library of Congress is undertaking a similar effort (Markoff and Wyatt, 2004). Nor is the plan without controversy, as publishers have expressed concern about copyright violations, although most will likely opt in to it (Butler, 2005). An ongoing bibliography concerning this topic is maintained at:
http://www.escholarlypub.com/digitalkoans/2005/10/25/the-google-print-controversy-a-bibliography/

Google continues to roll out additional tools that challenge the role of academic libraries (MacColl, 2006).

Butler, D. (2005). Publishers irritated by Google's digital library. Nature, 433: 446.
Giustini, D. (2005). How Google is changing medicine. British Medical Journal, 331: 1487-1488.
Greenwald, R. (2005). And a diagnostic test was performed. New England Journal of Medicine, 353: 2089-2090.
Henderson, G. (2005). Google Scholar: a source for clinicians? Canadian Medical Association Journal, 172: 1549-1550.
MacColl, J. (2006). Google challenges for academic libraries. Ariadne, 46. http://www.ariadne.ac.uk/issue46/maccoll/.
Markoff, J. and Wyatt, E. (2004). Google is adding major libraries to its database. New York Times. December 14, 2004. http://www.nytimes.com/2004/12/14/technology/14google.html.
Steinbrook, R. (2006). Searching for the right search - reaching the medical literature. New England Journal of Medicine, 354: 4-7.

(4/1/03) Controversy and tension over public databases continues is not limited to biomedicine. In 2002, the Department of Energy closed down the PubScience database and its associated Web site. PubScience was a bibliographic database covering a wide spectrum of chemistry and physics literature. PubScience was to the physics and chemistry communities what PubMed is to the biomedical community. The closure was felt by some to be in part due to the pressure of commerical interests. An overview of the issues can be found in a statement published by the American Library Association (Sheketoff, 2002).

Sheketoff, E., Baish, M., et al. (2002). PubSCIENCE: A Unique and Needed Scientific Resource. American Library Association. http://www.ala.org/washoff/pubscience.pdf.

(4/1/03) An overview of electronic journals in medicine was provided by Curran (2002). He reviewed the historical evolution of electronic medical journals and described how the electronic process improves their production and timeliness. He also claimed that publication bias against studies with negative results could occur with them, though does not really provide any data to support it being any different than publication bias with paper journals.

Curran, C. (2002). The medical journal meets the Internet. First Monday, 7: 6. http://firstmonday.org/issues/issue7_6/curran/index.html.

2.6.1 Benefits and challenges of electronic scholarly publication

(4/4/07) The growing effort devoted to unimpeded access to scientific information has taken on the name "open access" publishing. It is guided by the philosophy that access to scientific archives should be free and unimpeded, with other means used to finance the cost of scientific publishing. There is considerable debate over the merits of this approach to publishing, including discussion of its financial feasibility as well as the issue of who controls scientific literature. The typical solution proposed for financing is that the author pays, based on the notion that most research is funded by grants, and a small additional charge for publishing should not adversely effect their budgets. (In fact, most researchers with grants consider the time they spend writing papers about the research to be part of their salary time that is funded.) Most open-access journals usually have provisions for those who cannot afford the page charges, especially scienitsts from developing countries. A bibliography of open-access publishing is maintained by Bailey (http://www.digital-scholarship.com/oab/oab.htm)

The original open-access publishing effort in biomedicine was developed by Biomed Central (BMC), described in the book. Since the book was published, BMC has expanded to 176 journals (http://www.biomedcentral.com/info/authors/journaloverview) with over 15,000 scientific articles published. Most BMC paper are indexed in MEDLINE and archived in PubMed Central. Because BMC is a private company, it needs a business model where revenues exceed costs. The main source of revenue are the article-processing charges that authors must pay after their article is accepted for publication. Reflecting the reality of these costs, the charges have increased from the original $500 to nearly $1500 for the most expensive journals (http://www.biomedcentral.com/info/about/apcfaq). Individual journals can set their own rates. Most journals waive the charges for scientists who cannot afford them, making up for it by setting the rate for those who day somewhat higher.

One new innovation at BMC has been institutional memberships, where institutions such as universities pay a fee to join, in turn allowing any member of that institution to publish without paying the usual publishing charge. In the UK, the government recently reached an agreement to pay $170,000 per year to BMC that allows any UK researcher to publish in BMC without paying the article-processing charge. Now that several BMC journals have been in existence for more than two years, they are receiving IFs. Several BMC journals have IFs > 5.0, and most IFs have increased over time for longer-established journals (http://www.biomedcentral.com/info/about/faq?name=impactfactor).

Another new journal taking the open-access approach is the Public Library of Science (PLoS, http://www.plos.org/). The text notes that PLoS began as effort to gather signatures advocating open access to scientific archives. Under the leadership of Harold Varmus, former director of the National Institutes of Health, PLoS has developed six journals using the open access model. Its first journal, PLoS Biology, began publication in 2003, while PLoS Medicine launched in 2004. Since then have launched PLoS Computational Biology, PLoS Genetics, PLoS Pathogens, and PLoS Clinical Trials. One goal of the latter is to reduce publication bias for RCTs by making it easier to publish their results.

Most other journals have not opted for the open-access approach. Lancet has addressed the open access issue in its pages, noting that the up-front page charges may limit venues of publishing for resource-poor scientists (although BMC and PLoS pledge to waive fees for such scientists) and that open access threatens the survival of non-profit presses, such as university presses (Horton, 2003). The editor of JAMA has also taken a negative view, noting that the article-processing charges of BMC and PLoS may not cover the costs of the publishing process, meaning that their business models are not sustainable, especially for journals with low acceptance rates like JAMA at 8% (DeAngelis and Musacchio, 2004). (She also noted that PLoS is funded by a one-time grant.) She also expressed concern that this model may provide incentive for journals to publish more and, as a result, lower their quality. The publisher of BMC, on the other hand, retorted that journals still need to maintain their quality if they want to provide incentive for scientists to publish there (J Velterop, personal communication).

One journal is actually going in reverse from open publishing. Long hailed as an innovator by making its entire content free on the Web, BMJ began restricting access to non-research articles (abridged articles, news stories, letters, etc.) in 2005 (http://bmj.bmjjournals.com/aboutsite/subscriptions2006.shtml). The journal notes undertook this action because subscriptions to BMJ had fallen off, resulting in decreased revenue for the production of the journal (Delamothe and Smith, 2003).

One alternative that has emerged to the open-access movement, mainly from non-profit publishers (typically professional societies), is the Washington DC Principles for Free Access to Science (Anonymous, 2004a). These publishers reaffirmed their view that they maintain the copyright on their publications, but advocate a number of principles:

Selected important articles available free on-line when published
Complete contents freely available within months of publication
Complete contents available for free to scientists in low-income countries
Content available through on-line reference linking and major search engines

As such, articles older than 6-12 months are made freely available on most of the Web sites of these journals, which include JAMA, NEJM, and Annals of Internal Medicine.

A big event in the debate occurred in July, 2004, when the US House of Representatives mandated that the National Institutes of Health (NIH) develop a plan to increase access to reports of US taxpayer-funded research. It instructed the NIH to develop a plan by December 1, 2004 that would become effective by July 1, 2005. The legislation espoused the principle that all articles based on NIH-funded research would be deposited in the PubMed Central repository, with articles produced by any amount of NIH funding entering immediately and all others within six months. The legislation was based on a report by NIH Director Elias Zerhouni published in May, 2004 (Zerhouni, 2004a). An overview of this process has been maintained by Suber (http://www.earlham.edu/%7Epeters/fos/nihfaq.htm).

This legislation reflected other sentiments from around the world. A summary of various worldwide efforts to promote open access publishing was published earlier in 2004 by the European Molecular Biology Organization (Grivell, 2004). A report to the United Kingdom House of Commons promoted similar access (Anonymous, 2004a; Clery, 2004). Also in 2004, As a result of the legislation, the NIH released a request for comments (RFC) on the principles (Anonymous, 2004c). Zerhouni (2004b) summarized the process in the fall of 2004 and put forth an analysis of the cost. For the latter, he based his work on an estimate that 0.32% of grant funding is devoted to publication costs. Based on this estimate, the NIH already provides about $30 million in direct costs for publications through its funded research grants. Zerhouni asserted that adopting a plan to archive all publications in PubMed Central would add only another $2-4 million per year, since it would be built on top of the existing NIH information technology infrastructure. The RFC drew over 6,000 responses. Many were in favor, but both commercial and non-profit publishers raised concerns about the plan. Among the latter were public documents from the American College of Physicians (Tooker, 2004) and the New England Journal of Medicine (Drazen and Curfman, 2004).

In early 2005, the NIH released its policy on archiving scientific publications, adopting a voluntary policy whereby NIH-funded were strongly encouraged either themselves or via their publishers to deposit reports of their research into PubMed Central (PMC), an NIH article repository (described in Chapter 4), as soon as possible, and within 12 months after final publication (Anonmyous, 2005a). Not surprisingly, this political compromise pleased few participants in the debate, e.g., Washington DC Principles for Free Access to Science (Anonymous, 2005b). The debate may actually be a moot point, since only 4% of researchers have voluntarily submitted their papers to PMC. The report acknolwedges one ongoing concern that the article authors submit represents the final version they submitted to the journal, and may undergo subsequent editing by the journal editors.

Now that the open-access movement has been around for several years, overviews and research about it have begun to emerge. A number of overviews lay out the basic issues (Anonymous, 2005c; Clarke, 2005; Funk, 2005; Albert, 2006). Although some articles in the digital library literature express great enthusiasm for it as a convergence with the movements to open-source software and open science (Willinsky, 2005), others maintain that a subset of researchers and librarians who are strong advocates are no match for the power of both the commerical and non-profit publishers (Law, 2006). The latter notes that even though researchers and librarians who advocate open-access pubilshing have ambivalence about Google for serious research, the search engine may be an unwitting ally in efforts to open acccessibility to electronic literature.

Schroter and colleagues have published three studies surveying the knowledge and attitudes of authors toward open-access publishing. One study assessed authors who submitted papers to BMJ (Schroter and Tite, 2005). Most expressed support for the idea of open-access publishing, but few reported they had submitted to such journals. There was a strong sentiment that the quality of the journal influenced their submission decisions rather than the publishing model. Another study asked authors' opinions of a hyopthetical decision by BMJ to restrict access to subscribers for all of its on-line content,including research articles (Schroter, 2006). Only 14% of those surveyed said they would be much less likely to submit in the future if such restricted access were implemented, although two-thirds said it would diminish their view of the journal. A final study assessed author attitudes from BMJ and two other journals (Schroter and Tite, 2006). Before the survey was administered, less than half were familiar with the terms "open access" and "author pays." Only 10% had submitted to such journals. Open-access journals were viewed with skepticism: 27% thought they had lower impact factors and 46% believed that anyone who paid could get published (i.e., not having peer review). Slightly over half said that open access had low or no priority in their decisions on where to submit, and two-thirds said they would prefer to submit to a subscription-based journal.

Another study surveyed senior authors and found that while knowledge about open-access publishing is still scant, with 18% knowing "a lot" or "quite a lot" and 81% having some awareness of it (Rowlands, 2005). About 29% claimed to have published in open-access journals, though the researchers warned that many people interpreted open access to mean free availibility at their own institution, which could of course be the result on the institution having a subscription. These researchers also believed that article downloads were a better measure of the usefulness of research than citation counts. They also greatly appreciated electronic tools for tracking down references, rating physical libraries quite low as a place to find them.

A stuanch supporter of open access is the European Commission, wihch has published a report (Dewatripont, 2006) and policy statement (Anonymous, 2007) advocating wider access to published scientific literature, especially that funded by governments. This has led to Internet-based petitions calling for support of open-access publishing, including in the United States (www.publicaccesstoresearch.org).

Albert, K. (2006). Open access: implications for scholarly publishing and medical libraries. Journal of the Medical Library Association, 94: 253-262.
Anonymous (2004a). Washington D.C. Principles For Free Access to Science - A Statement from Not-for-Profit Publishers. Washington, DC, Washington D.C. Principles For Free Access to Science. http://www.dcprinciples.org/statement.pdf.
Anonymous (2004b). Scientific Publications: Free for All? London, UK, Science and Technology Committee, House of Commons. http://www.publications.parliament.uk/pa/cm200304/cmselect/cmsctech/399/399.pdf.
Anonymous (2004c). Enhanced Public Access to NIH Research Information. Washington, DC, National Institutes of Health. http://grants1.nih.gov/grants/guide/notice-files/NOT-OD-04-064.html.
Anonymous (2004d). Open access journals proven to compete on quality. London, UK, Biomed Central. http://www.biomedcentral.com/info/about/pr-releases?pr=20040624.
Anonymous (2005a). Policy on Enhancing Public Access to Archived Publications Resulting from NIH-Funded Research. Washington, DC, National Institutes of Health. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-05-022.html.
Anonymous (2005b). Not-for-Profit Publishers Call New NIH Rule a Missed Opportunity. Washington, DC, Washington DC Principles for Free Access to Science. http://www.dcprinciples.org/nih_rule.htm.
Anonymous (2005c). Open access in medical publishing: trends and countertrends. Canadian Medical Association Journal, 172: 149.
Anonymous (2007). Scientific information in the digital age: access, dissemination and preservation. Brussels, Belgium, European Commission. http://ec.europa.eu/research/science-society/document_library/pdf_06/communication-022007_en.pdf.
Clarke, M. (2004). Open sesame? Increasing access to medical literature. Pediatrics, 114: 265-268.
Clery, D. (2004). U.K. lawmakers urge prompt access to published papers. Science, 305: 458-459.
DeAngelis, C. and Musacchio, R. (2004). Access to JAMA. Journal of the American Medical Association, 291: 370-371.
Delamothe, T. and Smith, R. (2003). Paying for bmj.com. British Medical Journal, 327: 241-242.
Dewatripont, M., Ginsburgh, V., et al. (2006). Study on the economic and technical evolution of the scientific publication markets in Europe. Brussels, Belgium, European Commisssion. http://ec.europa.eu/research/science-society/pdf/scientific-publication-study_en.pdf.
Drazen, J. and Curfman, G. (2004). Public access to biomedical research. New England Journal of Medicine, 351: 1343.
Funk, M. (2005). Open Access - A Primer. Chicago, IL, Medical Library Association. http://www.mlanet.org/pdf/resources/oa_primer_mfunk.pdf.
Grivell, L. (2004). Access for all? EMBO Reports, 5: 222-225.
Horton, R. (2003). 21st-century biomedical journals: failures and futures. Lancet, 362: 1510-1512.
Law, D. (2006). Delivering open access: from promise to practice. Ariadne, 46. http://www.ariadne.ac.uk/issue46/law/.
Rowlands, I. and Nicholas, D. (2005). New Journal Publishing Models: An International Survey of Senior Researchers. London, England, Centre for Information Behaviour and the Evaluation of Research. http://www.ucl.ac.uk/ciber/ciber_2005_survey_final.pdf.
Schroter, S., Tite, L., et al. (2005). Perceptions of open access publishing: interviews with journal authors. British Medical Journal, 330: 756.
Schroter, S. (2006). Importance of free access to research articles on decision to submit to the BMJ: survey of authors. British Medical Journal, 332: 394-396.
Schroter, S. and Tite, L. (2006). Open access publishing and author-pays business models: a survey of authors' knowledge and perceptions. Journal of the Royal Society of Medicine, 99: 141-148.
Suber, P. (2004). NIH Public-Access Policy Frequently Asked Questions. Earlham College. http://www.earlham.edu/~peters/fos/nihfaq.htm. Accessed: April 3, 2005.
Tooker, J. (2004). ACP Comments on Proposed NIH Public Access Policy. Philadelphia, PA, American College of Physicians. http://www.acponline.org/hpp/nih_open.htm.
Willinsky, J. (2005). The unacknowledged convergence of open source, open access, and open science. First Monday, 10: 8. http://www.firstmonday.org/issues/issue10_8/willinsky/.
Zerhouni, E. (2004a). Access to Biomedical Research Information. Bethesda, MD, National Institutes of Health. http://library.cpmc.columbia.edu/hsl/eres/countact.cfm?resourceid=5673.
Zerhouni, E. (2004b). NIH public access policy. Science, 306: 1895.
Zerhouni, E. (2006). Report on NIH Public Access Policy. Bethesda, MD, National Institutes of Health. http://publicaccess.nih.gov/Final_Report_20060201.pdf.

(4/11/06) The book noted that some journals, most notably BMJ, have adopted a electronic-long/paper-short (ELPS) apporach. A sample of authors and readers were given three versions of articles to assess for their formatting preference: conventional scientific version, ehnanced-abstract version, and journalistic version (Muller, 2005). The latter was essentially a narrative of the article that described why the research was done, how it was carried out, and what the results showed. Authors expressed a strong preference for conventional version (56%) followed by the journalistic version (34%) and the enhanced-abstract version (27%) (all differences statistically significant). While readers still preferred the conventional version (42%), they did so with less strength (though still statistically significant), and they were nearly equally divided between the journalistic version (28%) and the enhanced-abstract version.

Mullner, M., Waechter, F., et al. (2005). How should abridged scientific articles be presented in journals? A survey of readers and authors. Canadian Medical Association Journal, 172: 203-205.

2.6.2 Quality of health information on the Web

(4/10/05) One of the new information resources achieving prominence since the publication of the book (described in the Chpater 4 update) is Wikipedia. It is mentioned here because a highly publicized study in Nature compared the accuracy and comprehensiveness of scientific topics in Wikipedia and the venerable Encyclopedia Brittanica. As noted in the Chapter 4 update, Wikipedia has a distributed authorship where anyone can update an entry. This of course is open to abuse, with the most recent example suggesting the involvement of John Seigenthaler, a former assistnant to US Sen. Robert Kennedy, being involved in the assassination of him and his brother. An outraged Siegenthaler called this "character assassination" (Seigenthaler, 2005) and this led to Wikipedia requiring posters to register and tracking their postings (Kornblum, 2005).

The Nature study compared 42 topics in Wikipedia and Encyclopedia Brittanica, removing the text to obfuscate its source and sending it to experts in the respective fields. A total of eight serious errors were found, four in each topic. However, there were more "factual errors, omissions, or misleading statements" in Encyclopedia Brittanica (162) than Wikipedia (123) (Giles, 2005). More detailed information about the study was provided in a supplementary document on the Nature Web site (Anonymous, 2005). Not surprisingly, Encyclopedia Brittanica took great exception to the study, calling it "fatally flawed" (Anonymous, 2006a) Also not surprisingly, Nature rebutted the claims (Anonymous, 2006b).

Nature apparently has confidence in Wikipedia, because it uses the on-line encyclopedia, as many do, for definitions to words that appear on its Web site (Blackman, 2006).

Anonymous (2005). Supplementary information to accompany Nature news article "Internet encyclopaedias go head to head". London, England, Nature Publishing Group. http://www.nature.com/nature/journal/v438/n7070/extref/438900a-s1.doc.
Anonymous (2006a). Fatally Flawed - Refuting the recent study on encyclopedic accuracy by the journal Nature. Chicago, IL, Encyclopedia Brittanica. http://corporate.britannica.com/britannica_nature_response.pdf.
Anonymous (2006). Encyclopaedia Britannica and Nature: a response. London, England, Nature Publishing Group. http://www.nature.com/press_releases/Britannica_response.pdf.
Blackman, S. (2006). Nature has Wikipedia in its cites. The Scientist, 20(2): 18-19. http://www.the-scientist.com/article/display/23079/.
Giles, J. (2005). Internet encyclopaedias go head to head. Nature, 438: 900-901. http://www.nature.com/nature/journal/v438/n7070/full/438900a.html.
Kornblum, J. (2005). It's online, but is it true? USA Today. http://www.usatoday.com/tech/news/techpolicy/2005-12-06-wikipedia-truth_x.htm.
Seigenthaler, J. (2005). A false Wikipedia 'biography'. USA Today. http://www.usatoday.com/news/opinion/editorials/2005-11-29-wikipedia-edit_x.htm.

(4/10/06) Despite the growing ubiquity of the Web, there are still concerns about the quality of information, yet there appears to be little that can be done outside of user education. Nonetheless, there have been a number of new research findings.

One continuing issue is how well quality measures for on-line health information can be used. Bernstam et al. (2005) have identified 273 distinct tools pruporting to help consumers assess the quality of health information on the Web, yet only seven met the usablity criteria of having evaluation criteria publicly available, having 10 or few items, and having elements that could be objectively evaluated. In a recent study, Bernstam et al. (2005) found that a set of 22 of these measures, when used in their original form, led to widely divergent reviews. However, when they were more precisely defined, 18 could be assigned reliably by medical experts.

Eysenbach and Kohler (2004) attempted to measure how often consumers use the Web for health-related searches. While previous research looked at how often people searched for health information one or more time, this study attempted to measure how many search-engine queries were actually health-related. They found about 4.5% of queries to the Metacrawler search engine were on health topics. The most common topics included

Health care services and organizations - 9.6%
Medications - 8.1%
Diet, nutrition, and weight loss - 6.7%
Infectious diseases - 5.9%
Cancer - 5.2%
Rheumatology - 5.2%
Women's health (excluding pregnancy) - 5.2%
Addiction - 4.4%
Mental health - 4.4%
Cardiovascular - 3.7%
Pregnancy and fertility - 3.7%
Dermatology - 3.7%
Natural/alternative medicine - 3.7%

A number of publications continue to promote frameworks and instruments for assessing quality of information on