A look at the world of published scientific papers and the issues around them.
I can think of no better lie than a lie wrapped in scientificism, polysyllabic words, and garnished with a bow of lab coats.
My first run-in with a rat in a lab coat well over a decade ago. The rat was the CEO of a medical device company. He had invented a medical device to help cure a skin disorder. The device had several patents and several more were pending. The evidence of the efficacy of the device for treating this skin condition came from several different published scientific papers. Further, the company had received a grant from the United States Department of Health and Human Services to continue it’s research, to document the efficacy of the device, and to look into other possible applications for the device.
My client was of the opinion that, based on the patents and the scientific research — that other than some sloppy legal issues — he was going to fund the company’s commercialization to the tune of 11 million dollars. I agreed with his conclusion and his approach to the company. I scheduled a visit to the company to conduct the final phase of due diligence: a visit to the research and manufacturing facilities.
When I arrived, I saw what purported to be the manufacturing facility and company offices. It was a typical set-up – okay offices and a rough research area that appeared at one time to have been a big glass-enclosed conference room. Nothing seemed out of place. Entrepreneurs work in the space available, rarely the best space. The principal of the company was odd. He would not leave my side and kept asking what I was looking at, what I was looking for, what I needed to know. Not only was his behavior annoying, but he reeked of fear. He stood very close to me and looked at everything I looked at, no matter how I tried to put him at ease. After a short tour of the facility, he put me in a side office with two chairs and a table full of documents. The trigger came as we sat down. He said, “These are all of the documents, all of the documents that you need to see.” I asked, “But what if I want something else, something not contained in these documents?” His clear response was, “There are no other documents you will see; these are the only documents you need.”
My due diligence alarm bells went from 10 to 100 in that instant. His actions were to create a classic a knowledge wall put up by someone trying to deceive. Someone who gives you the information they choose to provide and isolates you from other information. So, I asked, “What should I look at first?” The deceiver will typically aim you right at his best-constructed bits of disinformation. So you know right off where the problem rests; you know that as a due diligence professional, you must ferret out the information. I expected to see the financials or projections or testimonials. But to my surprise, he handed me three ring binders with all of the published papers that demonstrated the efficacy of the device.
All of my training and experience told me this was the problem. But all of the research had been done by third parties. It all looked thorough and well-documented. So what did I do? I sat and I read. I read the papers and the citations, word for word, while he perspired and stared at me. I took many notes and each time I left to get a glass of water or visit the restroom, he read through my notes. He was so nervous he would forget to replace my pen or the pad of paper where they had been when I left. In the early afternoon, he started to fade. He began to look haggard, and his personality went from over-anxious to annoyed. One cannot maintain a state of hypervigilance for a long period of time. The adrenalin and caffeine were all spent, and he was tired. We are at our worst when we are tired, and he was no exception. Now I was just waiting for the event. I did not know what the event would be; but I knew something was going to happen. Finally, he stood up, politely thanked me for my time, calmly took all of my notes, and asked me to leave. So I did.
The $11 million in funding died ten minutes later after I placed a call to the investor. My assignment was over. But, I am a curious person. What was he hiding? The investor sent me copies of the studies he had been given – and I noticed something – the research papers he had did not come with the citation end-notes. The citations had been removed; they had been cut from the documents the company had shared. As this was in the days before the internet of everything, I went to the local university library and pulled all of the papers to check their citations. Several of the papers had cited the same research. This is not unusual in a narrow field of research; so I pulled the cited papers. All of these research papers, two cycles of publishing removed, cited the same seven papers as their foundation. All seven of those papers were authored by researchers we were unable to locate, fictitious scientists. What we did find was that all seven of these papers had in common the same juror for publication. The juror was our entrepreneur scientist.
It was utterly brilliant. He had offered to be a juror in select fields of tropical dermatology for many different publications. As these publications always need jurors — and as a volunteer is the best sort of juror — his offer of assistance was accepted by the publications in which the seven papers appeared; and I am sure by other unknown publications. Once ensconced as a juror, he set about to submit papers he had authored, reporting on research he may or may not have done; and as he was the juror of his own papers, all of the papers were published. The second step was getting the next layer of papers published. He did this by helping people all over the world write papers citing the prior work; and he would then see that their papers were published. After all, he was one of the jurors. The final round of publishing funneled right back to him to assemble the research from others, the ones he had encouraged, to backstop his papers to be published. By this time, he no longer needed to be a juror as the research was well-documented in many other publications. His papers became for a literature review. It was truly a brilliant scheme, well executed. Did his device work? Who knows? Maybe. Maybe not. But taking money based upon faked information is a fraud. A go-to-jail type of fraud. What happened to him and the technology? I have no idea.
A few years later — I think it was 2001 — I picked up a book called “A Rum Affair, a true story of botanical fraud” by Karl Sabbagh.
The tale, in short, is that a fellow by the name of John Heslop Harrison proposed that certain species of plants on the islands off the west coast of Scotland had survived the last Ice Age. John Raven – an amateur botanist thought this was utter rubbish as the ice shield had extended well beyond these islands. Harrison’s response was utter disdain for the amateur; and to show Raven up, Harrison took folks to see the plants on Rum Island. Most concluded that the plants had been transported and transplanted there by Harrison himself, and Harrison was caught in an academic and scientific fraud. What was even more illuminating for me was the author’s discussion of how many in the scientific community carry on this practice of fraud and chicanery to the present.
As a due diligence professional you are always looking through the glossy narratives. You are looking for the foundations of the claims, either those made overtly or those assumed. When you are seeing that even the evidence before you, on which you are basing the choice of whether to invest millions, possibly even risk lives, can be manipulated, handcrafted to order, to fit the fraud. This is, and continues to be, a real problem.
VIPs – (Very Impressive People) with more letters after their name then their name contains — all crow about so-called evidence-based methodologies, those based upon scientific research. That choice should be made scientifically and should not be based on pure guesswork. By basing our decision making on science, we reduce or eliminate the risk of getting our choices wrong. After all, the choices are supported by science!
Very Impressive People with documented research present a very powerful argument. After all, we are told in school that science is about hard facts and discovering the truth about things and nature. Just witness the debate on climate change. Some have done research demonstrating that the climate is warming; some have done research showing the underlying numbers were fiddled with, and the climate is not warming. Some say a warming planet is bad and terrible things will happen; some say it is good as it will increase the number of arable acres for crops. The point made is that even in science – where facts are important – there is great debate on what is and is not good science. (Unless, of course, the science does or does not support your point of view – or your grant chances.)
So what are some common errors in making choices based upon science?
• Insufficient Sample Size.
Many papers in medicine are published with sample sizes of less than 100 patients. Add to this the unknown background of the people the sample, their ages, their genders, and even a small sampling bias, and results that look statistically significant on paper may be highly inaccurate. One gets fooled by tactical noise.
• Applicability Creep
Allowing a finding in one field of science to creep outside of the original parameters of the research. For example, many studies on rehabilitation for adults are being cited in support of using the same process for children. The bodies of adults and children are different. Bones, muscles, healing ability, growth plates all play a significant role in a child as opposed to an adult. Thus, by clear reasoning, the outcomes for adults and children will be different. While the studies may be suggestive, they are not to be considered indicative.
• Cherry Picking
Like anything thing else in life, one can cherry-picks studies that agree with your point of view and ignore the others. It is frustrating to note that we learn more from our failure then our success and that failures must also be considered and studied.
• Unable to be replicated
Much research is done and reported but when others attempt to replicate the work, they are unable to do so. Be it cold fusion, simple tissue staining, or mystery data sets, this happens a great deal. If the research cannot be replicated, it should not be considered as fact, only as a curiosity.
• Paper authored by a program
It appears that some papers are being authored by a computer program. The program generates utter nonsense in full sentences that are so convincing that several of the papers so generated have been published. Here is a link to one such paper generator
• Conflict of Interest
A large volume of research in the world is sponsored. It can be sponsored by governments, by foundations, by private parties or by companies with a vested interest in the outcome. In such cases, the funding can sway the results or the editing of the research hypothesis. The origin of the funding can create a significant bias for any research project. The funding origin does not always create problems; but knowing the source of funding for a project is important when considering the value of that research.
• Bad Math
It is worrying to say that the scientific community gets the language of their profession wrong. Typical errors may include a failure to understand statistical significance, errors in data mining, incorrect risk measurement, errors in sampling, miscalculation of confidence intervals, misunderstanding randomness, and not knowing the difference in difference. The rate of math errors is high. In some fields, the error rate exceeds 40%.
• Circle Citing
This is also known as networks of citation. Original research is done, and then other researchers begin citing the original research. Papers gain intellectual weight and gravitas each time they are referenced. Simply put, the number of times a paper has been cited influences the perception of how true the research results are. To promote their work, laboratories engage in organized circle citation. A research paper is published, and the laboratory then encourages others within the laboratory to reference the paper. Some laboratories will go so far as to pay outside authors to reference their papers. These networks of citation act like a focused lens, making a selected paper seem bigger and more important than it might actually be. The practice of circle citations engenders an engineered bias in the discipline through incremental misstatements and overstatements. With tens of thousands of journals and an estimate of over 900,000 scientific papers published per year, it is not hard to hide circle citing within all of that noise.
• Reproducibility
It is not scientifically significant if a study cannot be reproduced. Pons and Fleischmann learned about this when other tried to reproduce their results on cold fusion and other have learned the sting of their irreproducible results. Brian Nosek and a team took 100 psychological studies from 2008 from top journals and did their best to reproduce the results. Here is an excerpt from their abstract.
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Think about this very hard. Only 39% of the studies were judge to have been replicatable.
The idea in science is that research is conducted, results and methodology are published and the collegial world of science is open to challenge the results and the methods. But that assumed collegiality is bunk. For our experience a vast majority of researchers when contacted reacted from passively hostile to openly hostile to the idea of replicating their study. The hostility appeared to relate to us challenging their beliefs, their credibility, or most importantly their ability to get grants and funding.
If a scientific study has not been replicated, it is not yet scrience.
• Recalled and Retracted
Published papers are sometimes recalled. The papers are retracted for many reasons: disputes, unreproducible data, fraud, failures in procedures, manipulated data, unmentioned conflicts, misconduct in research, ethical issues, lack of scientific soundness, plagiarism, and rigged peer reviews, to mention just a few. Sometimes a paper is retracted because it was published without the knowledge of a named author. There can even be the honorable and honest self-retraction.
How bad can it be? I refer you to my heroes and my-go to site for retractions. The top 10 scientists have a total of 626 retracted papers. One knucklehead has over 180 retracted papers; and it looks like he is still publishing!
But wait — as they say on television – -there’s more. One would assume, as I did, that once a paper has been retracted, that paper is dead. No, my assumption was dead wrong. Some papers rise from retraction as unstoppable zombie bunnies. These zombified papers continue to be cited long after retraction. For example, we have all heard the popular myth correlating vaccination to autism. This legend has its foundation in a retracted paper, published by The Lancet, authored by Adam Wakefield et al. The paper was retracted in 2010 and exposed as a fraud by the BMJ. Wakefield’s misconduct was judged so severe that on 28 January 2010, the General Medical Counsel of the United Kingdom issued a ruling stating that he (Wakefield) had “failed in his duties as a responsible consultant,” acted against the interests of his patients, and acted “dishonestly and irresponsibly” in his controversial research. The end result was that on 24 May 2010, Wakefield was struck off the United Kingdom medical register. Yet, his discredited paper is still cited, and in March 2016, the zombified paper became a dodgy documentary. Wakefield’s research, sadly, is not the most persistent of the zombies. One retracted paper was cited over 1,000 times with three-quarters of the citations occurring after the published paper was recalled!
There is a lot of fraud and bad information in the scientific data pool. Just because a paper has been published, just because it is peer reviewed, does not make its findings correct. Sad to say, we cannot stop at reading just the content of a scientific paper; we have to become more educated consumers. We need to know how it was funded. We need to know the sample size and the techniques used. We need to research prior articles and cited documents. We need to actively hunt for contradictions and incongruences. And we need to read the fine print at the back and the bottom and check that too.
When you find something is utterly too convincing to be taken seriously, you should check to make sure the rat has fur and not a lab coat.
1) Here is a link to a automated paper generator https://pdos.csail.mit.edu/archive/scigen/
2) Research Gate Web Page Link https://www.researchgate.net/publication/281286234_Estimating_the_Reproducibility_of_Psychological_Science
3) www.retractionwatch.com The website for the work done by the amazing professionals at Retraction Watch.
4) http://www.bmj.com/content/342/bmj.c5347 link to the British Medical Journal for more detail on the matter.
