What is the problem with waitlist control groups in psychological research?

A study is published showing that a certain type of therapy is very effective for depression.

But how well do these results match what patients actually experience?

Many different factors determine the answer to that question.

The type of control group the researcher chooses is one of them. Several studies in the last decade have shown that waitlists are a commonly used control group in psychological studies and can result in the treatment appearing overly effective (Norwegian article).

How does that happen?

Why have a control group?

In a series of articles, we put the spotlight on the field of psychology. How well does it really work? Published so far:

• Why do researchers disagree so much about how well psychotherapy works?
• Can patients get just as well by talking to someone who is not a psychologist?
• Can teaching children to talk about their emotions contribute to their mental well-being as adults?
• Society is spending more and more money on mental health. Why aren’t we getting any better?
• Why are there 500 different types of psychotherapy?
• “I've wasted enough time going to therapy”
• Psychology folklore or science? Uncovering facts about repressed memories

For readers with mental health problems:

In this series, we ask questions about research into psychotherapy. Researchers disagree, and several speak out critically. But they do agree on one thing: if you have mental health issues, your chances of recovery are generally better if you go to therapy. And if you are getting treatment that you feel helps you, there is no reason to stop.

Researchers who study a particular type of treatment on humans need to have something to compare it to. For this they use a control group.

The reasons for control groups are several.

First of all, illness often varies over time.

In some cases, such as in patients with moderate depression, many patients get better on their own. In other cases, patients may feel worse in some periods and better in others. It is not unusual for people to see a doctor – and join studies – when they are going through a bad period.

If patients improve during the study, it might therefore not have been due to the treatment. They may have gotten better on their own.

Placebo effect

Second, we know that getting treatment in itself often leads to improvement. This is known as the placebo effect in medicine.

• You can read more about the placebo effect and its important role in medicine in the Norwegian article: The placebo effect: Fra mystisk magnetisme til å utnytte kroppens egne evner (The placebo effect: From mysterious magnetism to utilizing the body's own abilities)

If you test a treatment and patients get better, it is not necessarily the treatment method itself that had an effect. The result could also be due to a general placebo effect that would occur with any treatment.

However, both of these problems can be avoided by including a control group in the study.

By comparing the patients who receive the treatment you are studying with a group that does not receive it, you will be able to determine the effect of the treatment itself.

Several types of control groups exist, and which one you choose is critical for the reliability of your answers.

Waitlist controls

The best response is often achieved when the control group receives something that appears to be a real treatment, such as supportive conversations with healthcare personnel.

Most studies in psychology do not use this type of control group, however.

By far the most commonly used control method is to compare patients who receive treatment with people who are on a waitlist for treatment. This method compares getting treatment with not getting treatment.

It can provide interesting information.

You can find out if the patients just got better on their own, for example. If you see roughly the same improvement in the treatment group and the waitlist group, it is probably due to a naturally occurring improvement.

The problem is that the waitlist control condition does not provide any information about the placebo effect.

Unclear whether effect is just a placebo effect

After all, patients on the waitlist do not receive any treatment – and therefore no placebo effect either.

When the treatment group becomes healthier than the waitlist group in this type of study, you cannot know whether the improvement is due to the treatment or simply due to the placebo effect.

Any measure – such as singing in a choir or talking to a friend – could thus conceivably result in improvement compared to the waitlist group.

Exaggerated effect

The group receiving treatment thus has a high probability of getting better than the waitlist group, regardless of whether the treatment actually works. This type of study can also exaggerate the impact of a treatment that in reality only has limited effectiveness.

In this way, waitlist studies could possibly distort the picture of how well different treatments work, according the authors behind a 2022 research summary for the Cochrane Library.

Some researchers have argued in recent years that the problem could be even greater. Some studies suggest that being on a waitlist can actually have a negative effect on patients and may act like a ‘nocebo’ – the opposite of a placebo.

The patients on the waitlist may either experience less natural improvement over time or actually get worse while they wait.

Waitlist as nocebo

“We have several good meta-analyses that would suggest that people on a waitlist have worse outcomes than those simply not receiving any treatment,” says Joar Øveraas Halvorsen, a clinical psychologist and associate professor at NTNU.

“This can contribute to artificially inflated results,” he says.

If patient health in the control group declines during the study, any other treatment will seem to work better – including treatment that isn't actually effective.

Halvorsen points out that using waitlists as a control group is still a contentious issue, but believes it is very important for researchers to be aware of these problems.

Setting the bar too low

Jan Ivar Røssberg, a psychiatry professor at the University of Oslo and senior physician in psychiatry at Oslo University Hospital, also believes there is reason to suspect that patients get worse while on a waitlist.

“I think the waitlist control method is setting the bar a little too low. Given what we know today, I would recommend using other types of control groups,” he says.

Another problem is that leaving patients on a waitlist for a long time is unethical. This means that studies using waitlist controls can only last ten to twelve weeks before the patients on the waitlist are also receiving therapy.

Such studies thus cannot determine whether the treatment has an effect over a longer length of time.

Disagreement about waitlist studies

The use of waitlist control groups is one of the reasons that professionals disagree so strongly about how well psychotherapy actually works. You can read more about this disagreement in the article: Why do researchers disagree so much about how well psychotherapy works?

Because of its weaknesses, some researchers believe that waitlist studies should not be included in research summaries that assess the effectiveness of psychotherapy.

But other researchers disagree.

Eliminating the waitlist studies will artificially weaken the effect found for psychotherapy, according to eight Norwegian researchers in a 2022 Norwegian article in Psykologtidsskriftet.

Disagree that people on the waitlist get worse

Some researchers also disagree that patients get worse from being on the waitlist, on average.

“That is incorrect,” says Ole André Solbakken, a psychology professor at the University of Oslo and one of the authors of the article in Psykologtidsskriftet.

He thinks the data tell us something else.

“If you look at the meta-analyses in the field, you can see that people on the waitlist most often get better.”

Sverre Urnes Johnsen, a professor and clinical psychologist at the University of Oslo and Modum Bad, also says that he knows of studies from Norway that show patients get better from being on a waitlist.

How results are interpreted

“Using the waitlist as a control group has its weaknesses, but the crucial factor is that you compare the treatment group with something,” says Johnsen.

He believes the most important thing is for researchers to be aware of how they use and interpret such studies.

Waitlist studies are easy to carry out and could be suitable when first studying a new treatment, for example. The results might indicate whether starting more thorough investigations would be worthwhile.

“But we can’t stop there and claim that we’ve found the solution,” says Johnsen.

“The next step is to put the theory into action and to set up a larger study where we compare the effectiveness of the best documented forms of treatment in the field. This means comparing active forms of treatment with each other.”

Effectiveness often based mostly on waitlist studies

The problem is that Johnsen’s proposed approach probably happens to a lesser extent than it should.

Beth Patterson from McMaster University and her colleagues wrote in a 2016 study that in reality, “the use of waitlist controls continues to be a mainstay” in assessing different forms of treatment.

Summaries from the previous year showed that more than 70 per cent of published studies on the treatment of depression and anxiety had been carried out using the waitlist control condition.

This means that some of the forms of treatment we offer might in reality be far less effective than the studies suggest, wrote Patterson and colleagues.

Two studies from 2022 – this one and this one – reinforce Patterson’s statements.

The authors of the research summary from the Cochrane Library suggest the same as Johnsen:

Perhaps waitlist control groups should only be used in the initial exploratory studies of a new treatment.

References:

Erlend Faltinsen et.al.: Control interventions in randomised trials among people with mental health disorders. Cochrane Database of Systematic Reviews, April 2022.

Falk Leichsenring et.al.:  The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta-analytic evaluation of recent meta-analyses. World Psychiatry, January 2022.

Ioana A. Cristea: The waitlist is an inadequate benchmark for estimating the effectiveness of psychotherapy for depression. Epidemiology and Psychiatric Sciences, June 2019.

John A Cunningham  et.al.: Exploratory randomized controlled trial evaluating the impact of a waitlist control design. BMC Medical Research Methodology, December 2013.

T A Furukawa  et.al.: Waitlist may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatrica Scandinavica, September 2014.

Beth Patterson et.al.:  The use of waitlists as control conditions in anxiety disorders research. Journal of Psychiatric Research, December 2016.

———

Read the Norwegian version of this article at forskning.no