By Jeffrey S. Barkin, M.D.

Dr. Barkin has disclosed that he has no significant relationships with or financial interests in any commercial companies pertaining to this educational activity.

If you are like most busy psychiatrists, reading journal articles is about as entertaining as filling out treatment authorization forms. The good news is that keeping up with the literature can be fun, if you have a systematic way of approaching it. In this article, I’ll describe an approach to reading the literature designed to derive the most useful information in the least amount of time.

1. Who funded the study? According to a recent report, studies whose authors have financial conflicts of interest are five times more likely to find the
sponsored drug effective than other studies (Perlis RH et al, Am J Psychiatry 2005;162:1957–1960). While industryfunded studies can be well-designed and
valuable, you will need to give the conclusions extra scrutiny.

2. Are the patients being studied similar to the patients you treat? Most randomized placebo-controlled trials have strict inclusion criteria. For example, typical antidepressant trials exclude patients with symptoms that are either too mild or too severe, patients with comorbid substance abuse, bipolar disorder, psychosis, or suicidality. One study concluded that patients who make it into research trials represent only about 20% of the patients that real clinicians actually treat (Zimmerman M et al, Am J Psychiatry 2005;162:1370-1372). Results based on such samples have low generalizability, also termed low external validity.

3. What type of study design is it? For research questions that compare two treatments, the randomized controlled trial (RCT) provides the most accurate results. In RCTs, patients are randomly assigned to two or more intervention groups, one of which has a known (or generally accepted) outcome (the control). For tests of new medications, the gold standard is the double-blind placebocontrolled RCT, in which at least one treatment group receives an inactive pill (the placebo), and neither the researchers nor the subjects know who is taking the placebo. Making a study double-blind ensures that both groups are affected equally by whatever positive and negative expectations the researchers and/or the subjects have about the treatment. Randomization helps to assure that treatment groups will not systematically differ
in ways that the researchers think might influence the effect of the treatment; lack of randomization leads to the possibility of selection bias.

In open label trials, there is no attempt at blinding. Both the researchers and the patients are aware of the treatments, creating opportunities for bias. For example, researchers might unconsciously rate patients as improving more than they really did. Likewise, patients might be imbued with high expectations of a promising treatment and might experience a larger-than-usual placebo effect. Open trials are helpful in identifying potentially effective treatments, but the history of psychiatry is full of treatments that looked great in open trials, but which then failed to outperform placebo in RCTs, such as Neurontin for bipolar disorder (Frye M et al, J Clin Psychopharmacol 2000;20(6):607-614) and BuSpar for augmentation of antidepressants (Onder E et al, J Affect Disord. 2003 Sep;76(1-3):223-227).

The cohort study is a way of doing a controlled trial without having to actually assign subjects to any groups. Here, two cohorts, or groups, are identified, one which received the treatment of interest, and one which did not. These two groups are observed prospectively or forward in time, studying the outcome under analysis for each group. A typical example of a cohort study would be a study of antidepressant use in pregnancy. Because of concerns of possible risks to the fetus of AD exposure, pregnant women are not randomly assigned to drug vs. placebo. Instead, researchers identify women who happen to have been prescribed ADs, and compare them with a group who did not. Since the women were not randomized to the two groups, they may differ from one another in important ways. For example, it may be that women who opted to receive ADs were more depressed than the other group. If the study finds that infants exposed to ADs have more neonatal problems, it would not be clear if the problems were caused by the medications or by the depression itself.

A case series is simply a description of a group of patients with a particular illness who have received a particular treatment. This is often retrospective, meaning that the author reviews old charts to extract information on a series of similar patients. Like open label studies, these reports are suggestive but not definitive.

4. What are the identified primary and secondary outcomes of the study? Studies are typically designed in order to assess one or two primary outcomes, such as percent change in the Hamilton Depression scale, rate of remission, or time to treatment discontinuation. These are generally chosen because they are the most clinically relevant measures. If the primary outcomes do not reveal a difference between two groups, the authors will move onto a number of less relevant secondary outcome measures. The more extensive this statistical “fishing expedition” becomes, the more likely they are to find a statistically significant difference by chance alone. For this reason, savvy readers will focus on the results of declared primary outcomes.

5. How did the study deal with patients who dropped out? Many research patients drop out for various reasons, such as adverse events or clinical worsening, and there are different ways to account for these. The most conservative is called LOCF, or last observation carried forward. Here, each subject’s last score is included, regardless of when they dropped out. As you can imagine, if a medication causes many early drop-outs, the LOCF method will tend to drag the final average depression score down, making the medication appear relatively less effective. This is precisely the kind of information we need to know as clinicians, because the ideal medication should be both efficacious and well-tolerated. By contrast, the weaker method of reporting results is called OC, or observed cases. Here, only the subjects who stayed in the study until the very end are counted, ignoring any and all drop-outs. Somewhere in between LOCF and OC is a complex statistical technique called MMRM, or mixed model repeated measures. Here, patients who dropped out are compared with similar patients who completed the study, and their scores are statistically extrapolated based on these comparisons.

6. Are the results both statistically and clinically significant? Studies that enroll very large numbers of subjects may report advantages of a medication that are statistically significant but of only marginal clinical significance. For example, the rates of nausea for one antidepressant may be 50% but “only” 45% for the competitor, a result reported as “statistically significant” but of dubious clinical significance.

TCPR Verdict:

Time to tackle your stack of articles!