How to Evaluate Studies on Marijuana Medically and Otherwise
©Leonard Frieling 2018 Lafayette, Colorado
We are bombarded by press touting studies which purport to show wonderful benefits provided by marijuana. We are similarly bombarded by studies which present alleged dangers associated with marijuana use. How do we evaluate studies? Do we want rely upon in debates and discussions; studies to parade at rallies which say what we want, regardless of the scientific validity of the work? Personally I am a fan of intellectual honesty first and politics second. Let the facts drive the politics. Let the science guide the law.
On politically charged subjects like marijuana and firearms for example, the disparity in study conclusions is radical.
How does someone tell how much weight should be given to a study and its conclusions? How do we evaluate studies? Scientists have specific mathematical tools for evaluating study reliability. For most of us, these technical validity instruments are beyond our skill sets. But there are many ways to help us evaluate studies without the experience and education needed to utilize the sophisticated tools available to scientist. This topic will be covered in a series of blogs, since it is complex and worthy of study.
First, is the study a “meta-study” or a “study?” A study deals with obtaining new data and analyzing the new data. A Meta-study is one in which no new data is obtained and no new experiments are done. Instead, previous studies are re-analyzed. The raw data is re-examined in light of the other studies being examined, and “new” conclusions are drawn.
Meta-studies, while not without value, should be viewed differently. First, while at first glance they may appear to be, and may be presented as new studies, they are not. They are in many ways, old news re-runs.
That leads directly to a major problem in trying to track marijuana science by reading the newspapers. It is common to see an old and frequently weak study be trotted out as “new.” In fact, GIGO, “garbage in garbage out” is the fact. A weak study does not become a strong study because it has aged.
Second, who did the study? Who financed the study? It is no surprise that the studies glommed onto by law enforcement organizations, some police chiefs, and others, were financed or conducted by, written by, or interrupted by law enforcement or by other groups who potentially have agendas. Science and politics should not be the same thing. While science may guide legislation, study outcomes should not be results-driven. A study should not be undertaken if the goal from the outset is to prove that marijuana is harmful, or that it is not harmful.
Finally, in the “Big Three,” studies will generally end with statements suggesting what future research is needed or suggested. While many like to say “we need to study this more,” the reality is that while additional study is almost always good, the implied lack of historical studies is frequently the opposite of the truth. Studies should lead to more questions. That does not mean that the study provides no answers or guidance.
For example, the oft-repeated chant “we need more studies” regarding marijuana medically and otherwise is true. What it implies, that we don’t have many studies already, is the opposite of the truth. For example, pubmed.gov is an index of studies in the medical fields. Some are available free, while others provide a free summary and the ability to purchase the full study. Search pubmed for “aspirin” and 63,000+ studies are listed. Search for “marijuana” and 29,000+ results are available on this site. This makes two points. First, those that argue “we haven’t studied marijuana enough” are ignoring the fact that there may be almost half as many studies as for aspirin. Additionally the sheer number of the published studies on pubmed for marijuana is huge. Of course we need more study. And we simply cannot ignore the vast body of what has already been studied. More studies are needed in astrophysics. Does that suggest we know nothing currently?
What is the population studied? Is it ten subjects or 10,000? Is it over a period of a month or a longitudinal study over decades or even over generations? What is the source of the population? Are they properly screened for drug history or does the study rely upon the subjects accurately reporting their drug use, past and present? At least two studies which keep surfacing in the press persist in claiming damaging mental health from marijuana use. Both studies focus on a population suffering from severe mental illness.
Final suggestion for this Part I: READ THE STUDIES. While some are quite challenging and involve complex use of statistical analysis and extremely challenging vocabulary, many are sufficiently accessible so that even without understanding every word, reading the study is well worth the effort.