Why digging up primary sources is important?

When writing a thesis, a chore that always takes more time than predicted is building the bibliography. Even with good software to manage your citations and references (EndNote, RefWorks, JabRef etc.), peppering your text with references and engaging in discussion with your sources takes time. (I often wonder how it was even possible to do research before ScienceDirect, BibTex and Google Scholar).

A researcher is a part of a network and a link in a chain. We build on other researchers’ work and provide a service of collating information from numerous sources, making interpretations and value-based judgments on the way. Some papers wind up being generally accepted as “canonical” in the field, these are the ones read by every journal club and quoted in every paper and thesis on the topic. Others are forgotten or live on as curiosities, mentioned for entertainment value or as “sign of the times past”.

Efficient use of databases and information sources is the bread and butter of research training courses. This is important in order to prevent problems of omission: not finding and quoting a directly relevant paper, where in the worst case, the research question has already been answered.

Sometimes, especially for undergraduate students, the task of looking up primary sources seems an unnecessary waste of time. They find a book that summarises the topic they are learning about, and find a handy summary of prior research. Then, the frequently asked question is, how do I quote Author A that was mentioned in this book by Author B? While there is indeed a way of quoting a quotation, answering this based on what researchers ACTUALLY do would be much more worrying. The “honest” answer, the way this works in real life (as much as research can be described as such) seems to be that you just quote the stuff not mentioning where you found it, and trusting Author B to have gotten both the content and the reference right. I mean, they have written a book/review article/original research paper, they must be good researchers, and know their stuff, right? Plus the editor/referee must have checked these things for errors, right?

Wrong-o.

I remember that while we devoured papers for our Music and Science seminars in Cambridge, we often spotted things that the authors got wrong – papers they should have read, what we thought were misinterpretations of data or other peoples’ results, or methodological errors in their analysis. But, human scientists produce papers that are not perfect, and any paper put through the mill of scepticism formed by a group of 10-15 researchers from different fields, it is going to come out with a healthy splodge of red ink on it, no matter if it is the celebrated paper published in Science by a group of luminaries.

But I never suspected that so much of what is printed is just blatantly wrong, mainly due to laziness. Reading these three papers make the trip down to the library feel less of a bother. The first paper is by Porrino, Tan and Daluiski, published in 2008 in the Journal of Hand Surgery (I notice that writing a post on the topic of misquotations makes me check my references for typos etc. so much more carefully…). Porrino and associates looked at a seminal study in hand surgery, and tracked down the papers where it was quoted (this is made possible by yet another extremely handy tool, the ISI’s Scientific Citation Index).

They learned that a whopping, staggering, nauseating, etc. 40.9% of the 154 articles where the original text was quoted contained at least one error. In 34% of the papers, at least one error was made that could be categorised as major, relating to major interpretation errors or errors regarding treatment recommendations etc. Here it is important to remember, that we are talking about medical journals and treatments that can be life-changing, not just academic debate on what kinds of conclusions and interpretations one could form on the composers gender identity based on the characters in the operas they write. Not saying the latter is not important or proper research, just saying that nobody is going to die because you misrepresent somebody else’s work on the topic.

Similar figures were obtained by Hol, de Jonghse and Niewenhuis in their study on how the “most referenced” study in the field of probiotics was treated by the academic community. The paper by Kalliomäki et al. in Lancet, 2001, had been quoted in 663 papers, of which the authors accessed 458. Again, going through the quotations they found that 29% of the papers had misrepresentations or errors. Interestingly, they replicated their analysis with two other papers, both less quoted, and found that the percentage of errors was smaller, but still intolerably high, at around 15%.

In Porrino’s paper, the authors found that review papers are more often wrong than papers presenting original research, but Hol et al. didn’t observe a similar trend in their data. In both studies the types of error were many and often non-trivial. For instance, saying that the probiotic prevents other allegic diseases than atopic excema, when actually the authors’ results were exactly the opposite, can be considered a major error.

Finally, in addition to these two quantitative analyses of misquotations of a single paper, a more entertaining study by Michel Paradis. He goes through literature from his own field, neurolinguistics, and finds both the “number and kind” of the errors “astounding”. He is in a good position to judge these things, as he wrote many of the papers that had been misquoted, misinterpreted or simply garbled. This is an entertaining read, but should help you calibrate your moral compass regarding quotations next time you are writing a paper yourself.

All this might provoke very uncomfortable questions, even such as whether the peer review system is broken, as it seems to fail the propagation of such errors. My take is that just as with democracy, it is not perfect but it is the best we’ve got. We all need to do a better job, as researchers and as teachers, to stop the bugs from spreading. I think most researchers, at any given time, have on their (virtual or actual) desks papers to write, and papers to review. Therefore springing from intention to action should be exceptionally easy.

ResearchBlogging.org

PORRINOJR, J., TAN, V., & DALUISKI, A. (2008). Misquotation of a Commonly Referenced Hand Surgery Study The Journal of Hand Surgery, 33 (1), 20-2000000000 DOI: 10.1016/j.jhsa.2007.10.007

Hol, J., de Jongste, J., & Nieuwenhuis, E. (2009). Quoting a landmark paper on the beneficial effects of probiotics Journal of Allergy and Clinical Immunology, 124 (6), 1354-2147483647 DOI: 10.1016/j.jaci.2009.07.047

PARADIS, M. (2006). More belles infidèles—or why do so many bilingual studies speak with forked tongue?☆ Journal of Neurolinguistics, 19 (3), 195-208 DOI: 10.1016/j.jneuroling.2005.11.002

2 thoughts on “Why digging up primary sources is important?

  1. Wow, just wow.

    Its not really that surprising, i suppose, given that in the course of my PhD i’ve come across many errors like this.

    Its shocking though.

    I have a feeling that it may relate to the pressures of publication and the norms of referencing, where you need to have some source for everything, even if its not correct.

    What would be interesting would be a demographic analysis of the kinds of authors, topics and papers where this sort of thing is more likely.

  2. Pingback: Quick Links | A Blog Around The Clock

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s