If you’ve ever conducted an evaluation of your tobacco cessation program, you may be familiar with the distinction between intent-to-treat and responder quit rates. In case you’re not familiar with these terms, read on. Not only will knowledge of these methodologies inform your future evaluations of all your health promotion programs, but it will help you:
- Identify the interventions that are best to offer your employees.
- Present outcome data on your program’s interventions — not just tobacco cessation, but interventions for any health risk — in a more credible manner.
- More readily identify “spin,” when it is applied to the success rate of treatments — for example, weight loss programs — in the lay press and in peer-reviewed journals.
Usually, in an employee wellness program, tobacco cessation programs are evaluated based on participant surveys conducted six months and/or one year after completion of the program. In the survey, the participants are asked whether they are still tobacco-free. (A few programs, especially pharmaceutical studies and some of the new so-called outcomes-based wellness programs, use cotinine tests.)
The two fundamental models for analyzing results are:
- Intent-to-treat, in which the denominator includes all the participants who started the program. This leads to calculation of a lower quit rate.
- Responder quit rate, in which only the participants who were reached for the survey and who responded are included in the denominator. This leads to a higher quit rate compared to the intent-to-treat rate.
Responder Analysis
Here’s an example: Let’s say 100 people participate in your smoking cessation class. One year after the class concludes, you attempt to survey the participants by phone, email, and snail mail. Fifty participants complete your questionnaire. Of those 50 people who complete the survey, 25 are still smoke free. Based on a responder methodology, your quit rate is 50%. That is, 50% of those who responded to the survey were smoke free.
Intent-to-Treat Analysis
Using those very same results, let’s calculate the intent-to-treat rate. You have 25 people who successfully quit, but now your denominator is 100 — the number of people who started the program (regardless of how many of them didn’t respond to your questionnaire or dropped out of the program). Your intent-to-treat quit rate is 25% — that is, you’ve established that 25% of the 100 people who started the program have quit.
Even if you used cotinine testing, you could still have an intent-to-treat methodology which counts everyone who started your program. The alternative would be to use as the denominator everyone you actually tested, disregarding those who dropped out of the program or were unreachable. In this case, your alternative is not called a responder quit rate — it may be called a per-protocol rate.
Neither the intent-to-treat rate nor the responder rate is incorrect. The intent-to-treat rate is more conservative. The responder rate more specifically measures the strict effectiveness of the intervention without regard, say, for adherence. It’s possible that the actual quit rate lies someplace in between the two calculations, as it’s entirely likely that some of the participants who did not complete your questionnaire did, in fact, quit smoking. On the other hand, when the percentage of participants not measured is large, the responder rate is almost certain to overstate the success of the program and sweeps under the carpet what is likely a large nonadherent population. Indeed, the response rate (percentage of participants who responded to the survey) must accompany any report of a responder quit rate.
When I present results of smoking cessation programs I oversee, I’m likely to present both rates, and to educate my audience about the difference. For quality improvement purposes, I prefer the intent-to-treat measures.
For a more detailed discussion of tobacco cessation (specifically, quit line) evaluation, including a detailed discussion of intent-to-treat vs. responder quit rates, see the North American Quit Line Consortium’s issue paper, Measuring Quit Rates (pdf).
Relying on responder or per-protocol analysis is a favorite strategy of weight loss marketers. Have you ever wondered how any weight loss intervention can boast high success rates, when all you have to do is look around (and read a few studies) to know that most don’t work? Weight loss programs use short measurement periods (a year or less) and per protocol rates to veil the fact that after 12 months, a large proportion of their participants have either dropped out or are in the process of relapse.
When you read studies or other reports of successful interventions, look for intent-to-treat methodology, or at least an acknowledgement of alternative methodology, as one sign of credibility.
