The Cyberonics brochures have been leaping into your mailboxes. “A new strategy for the challenge of treatment-resistant depression,” they proclaim.

Vagus nerve stimulation (VNS) is certainly new—but is it effective? It took the FDA a long time to make up its mind, but eventually it approved the treatment in May 2005. (For more details on why the FDA flip-flopped on the issue, see “FDA Approval of VNS,” this issue.)

In this article, we scrutinize the two pivotal studies leading to approval. But first, let’s briefly review the FDA approval process. Traditionally, the FDA requires at least two double-blind placebo-controlled trials showing that the treatment is statistically better than placebo. If you can brandish two studies with the required methodology and results, you’re in the market.

This is exactly how Cyberonics originally obtained approval for VNS treatment of refractory epilepsy in 1997. They submitted two randomized, placebo-controlled, double-blind trials, both of which showed a statistically significant advantage for the primary outcome variable -- reduction in seizure frequency (Epilepsia 1994; 35:616-626, and Neurology 1998; 51:48-55). Since the treatment was approved, over 30,000 epilepsy patients have received VNS and it appears to have long term efficacy and safety (Neurology 2002; 59(Suppl 4):S15-S20).

What about the depression trials?

The first of the two pivotal psychiatry trials was also a randomized, placebo-controlled, double-blind trial. Well…sort of!

VNS Study # 1: Randomized, controlled, “double-blind” trial. Here’s how the study went; we invite you to look at the methodology and be your own judge.

At 21 different sites in the U.S. and Canada, researchers recruited 222 patients with treatment resistant depression (TRD). These were very ill patients: the average length of their current depressive episode was about four years, over half had received ECT at some point, and they had tried an average of 16 different medications or other depression treatments during their lives.

The patients were randomly and blindly assigned to either active VNS (112 patients) or sham treatment (110). In the sham treatment, the VNS electrodes were surgically implanted but were never turned on. The treatment lasted for 10 weeks, during which patients were allowed to continue whatever antidepressants they were already taking. At the end, they were all rated with a series of standard clinical scales by raters blinded to the treatment.

Sounds pretty good so far. But was it truly “placebo-controlled”? Recall that placebo pills are the typical mechanism used in studies to ensure that patients don’t know whether or not they are receiving the new and exciting and (hopefully) effective treatment. Clinical trials can’t get rid of the placebo effect—this is always going to be active in humans receiving medical treatment. But study designers try to make sure that this mysterious healing effect works to the same degree in both the actual treatment arm and the comparison arm of a study.

Creating a placebo VNS treatment is much trickier than creating a placebo pill. This is because a working VNS device causes hoarseness and other sensations for most patients for 30 seconds every 5 minutes. Patients who get such symptoms are likely to figure out that they were randomized to VNS, which “destroys the blind” and calls the results of the entire study into question.

In the VNS epilepsy studies, researchers went to great lengths in order to “protect the blind.” How did they do this? By designing the study so that the control group did not receive inert sham VNS, but rather “low stimulation” VNS. Patients in the control group had their devices adjusted to the “minimum level consistent with a sensation of being stimulated.” Presumably, the level of stimulation was low enough not to be an effective anti-seizure treatment, but high enough to produce the same “amount” of placebo effect in the two study arms. And even with a convincing placebo, the treatment arm separated from placebo, and the FDA gave VNS the green light for epilepsy.

In contrast, in the depression studies, this placebo equalizing technique was inexplicably dropped from the study design. Control group patients got their implants all right, but the devices weren’t set at low stimulation—they weren’t turned on at all! This meant that the VNS patients had a clue that they were getting active treatment (55% of them reported noticing “voice alteration”), and in the sham group, where only 3% reported voice alteration, patients may have suspected that they were not getting active treatment. Thus, those patients assigned to active VNS were more likely to benefit from the placebo effect than the control group. We know from the antidepressant medication literature that the placebo response is powerful and may account for as much as 80% of overall treatment effects.

Basically, this meant that the deck was stacked in favor of VNS from the start.

So let’s get down to the results.

On the primary outcome measure (the Hamilton depression scale), the VNS group had a response rate of 15%, compared with 10% in the sham group, which was not statistically different. Three “secondary” outcome measures were also used, only one of which separated statistically from placebo.

Of course, there are different ways to spin these results. You could say that VNS is not effective, because there was no statistically significant difference on the primary outcome measure, and two of the three secondary measures. Alternatively, you could say, as the Cyberonics-funded authors of the study did, that “By all measures, VNS was associated with greater symptom reduction.” (Biol Psychiatry 2005; 58:347-354; the quoted statement is on page 353). By this they meant that if you ignore issues of statistical significance, and just compare response rates, VNS squeaked by sham treatment on every outcome measure (and we mean “squeaked”—for example, on the CGI (Clinical Global Impression) the response rates were 13.9% vs. 11.8%).

The Two Pivotal Studies of VNS for Treatment Resistant Depression

Study	Study Type	Statistically Sig. Difference in Response Rates?	Notes
Rush et. al.	Randomized controlled “double-blind” trial	HamD (Primary Outcome): No Diff. MADRS: No Diff. CGI: No Diff. IDS-SR-30: 17% to 7% advantage for VNS	Patients were probably not blinded to treatment, leading to overestimation of VNS effects.
George et. al.	Naturalistic study of VNS compared to a similar naturalistic study of “treatment as usual.”	IDS-SR-30 (Primary Outcome): No Diff.HamD: 27% to 13% advantage for VNS CGI: 34% to 12% advantage for VNS	Study was not randomized or blinded. VNS group received more personal contact than TAU group.

Rush et. al., Biol Psychiatry 2005; 58:347-354
George et. al., Biol Psychiatry 2005; 58:364-373

So the first pivotal study was extremely marginal, both in study design and results. What about the second pivotal study?

VNS Study #2: Naturalistic, “controlled” trial. The second VNS study failed to meet two of the methodological criteria usually required for FDA review: it was not randomized, and it was not blinded. It was, however, controlled…well, sort of!

Cyberonics had funded an observational study of VNS patients in which 202 patients who had participated in the acute controlled trial (described above) were invited to continue receiving VNS in an open study. The purpose was to see how patients did over the long term, both in terms of efficacy and safety. As with most open trials of a new treatment, many patients improved over time; but since this was an open trial, it could not be submitted to the FDA.

Once the disappointing results of the first study were analyzed, Cyberonics realized that they needed better controlled data from somewhere. Rather than going to the expense of conducting a new study, they came up with a much cheaper solution: they transformed the observational study (that had already been completed) into a controlled study.

How did they accomplish this? Well, in parallel with the naturalistic study, they had also enrolled a different group of 124 TRD patients who were given “treatment as usual” (TAU). Originally, this study was conceived as a way of quantifying how expensive it is to treat TRD patients in the community—data that would presumably be eventually used to argue that VNS is a cost-effective alternative (just our guess!). Since they already had all this data about outcome in TRD in their computers, they figured they could use this as a control group for their VNS study, and then submit the results to the FDA.

As slap-dash and jerry-rigged as this process sounds, it actually happened, and it worked, leading directly to FDA approval.

The results of this “sort-of-controlled” study looked great for VNS: VNS patients statistically outperformed TAU patients in scores on the self-report depression scale (the IDS-SR30) and in response rates on the HamD (27% vs. 13%).

Whether this study actually demonstrated anything at all about VNS’s efficacy is a topic that will likely be debated for years, and which we suspect will go down in the annals as one of FDA’s major embarrassments.

While we don’t have space to expound on all of the methodological problems, we can at least spotlight some of the “greatest hits.” You and your methodologically-minded colleagues can use these as debating points at your next psychopharm soiree!

Top four potentially “Fatal Flaws” of the VNS Comparison Study

1. Treatment groups were not randomized. Possible confounding effect: One group may have been more treatment-responsive than the other.


2. Treatment groups were not blinded. The VNS patients knew they were getting VNS, and the “control” group knew they were receiving TAU, not VNS. Possible confounding effect: VNS advantage may have reflected patients’ positive hopefulness and expectancy rather than VNS effect; similarly, raters may have been biased due to their own excitement about the potentials of a new treatment and issues of academic advancement and future research grants if results were positive.


3. The VNS group received more personal contact than the TAU group (the VNS group met with researchers 16 times over the course of one year; the TAU group met with researchers--for briefer sessions--only five times over the course of the year). Possible confounding effect: More personal contact may lead to disproportionate degree of placebo effect in the VNS group, erroneously inflating response rates.


4. Medication treatment was not controlled (medication changes were allowed at the discretion of research clinicians in both groups). Possible confounding effect: Either group might have received more effective medication adjustments.

Time will tell; VNS may indeed be effective for TRD. However, we believe that the FDA should not have approved VNS without its usual requirement of two positive double blind trials.

TCPR Verdict:
VNS for TRD: The jury is still out (someone tell the FDA!)