One of the most common knocks on health risk appraisals is that, by definition, they rely on self-reported data. Critics assume self-reported data biased and inaccurate. In fact, one of the respondents in the Shape Up employer wellness survey was quoted as saying, “Biometric screening is a prerequisite for any program. …Self-reported is pointless.” So the real question is not just whether self-reported data is valid, but how it compares to other sources of data, such as biometric screenings (and medical claims data, too), generally perceived to be the gold standard.
The true validity of self reported data is murky. Most of the research on self-reported data validity is more than 10 years old, was conducted in demographic sub-populations, and is inconclusive.
One study of almost 40,000 women, published in the Journal of General Internal Medicine in 2007, found that women underestimated their total cholesterol by 9.7 mg/dL, but that higher levels of self-reported cholesterol are strongly associated with increased risk of cardiovascular disease.
In fact, many studies have found that certain self-reported data is consistently either understated or overstated, allowing “correction factors” to be built to adjust the self reports for a more accurate assessment of risk.
A (now out-of-print) white paper by Summex Health Management was released in August 2005, before Summit was acquired by WebMD. The paper cited several studies that cast a positive light on self reported data for various risk factors. Examples of some of the cited studies:
- Validity of Self-Reported Height and Weight in 4,808 Epic-Oxford Participants, which concluded that self-reported height and weight are valid for epidemiological studies.
- Self-Report of High Cholesterol: Determinants of Validity in U.S. Adults, which concluded that “self-reported hypercholesterolemia should be used with caution..Specificity is consistently much higher than sensitivity.”
- Effects of Age on Validity of Self-Reported Height, Weight, and Body Mass Index, which determined that “self-reported height and weight can be used with adults younger than 60 years.”
- Validation of Self-Reported Chronic Conditions and Health Services in a Managed Care Population, which found that self reports are “reasonably accurate” for certain chronic conditions and for routine screening exams and can provide a useful estimate for measures of prevalence.
- Differences in Morbidity Measures and Risk Factor Identification Using Multiple Data Sources: The Case of Stroke, which found that analysis based on self report “can provide valid, useful information.”
- Accuracy of Self-Report of Mammography and Pap Smear in a Low-Income uUrban Population, which concluded that the accuracy of self report of mammography and Pap smear is relatively poor for medical practice but is acceptable in population surveys with appropriate correction for over-reporting.
Based on these and other studies, Summex ultimately concluded, “Self-reported health data is valid for effective use in population health management intervention.” Their review of the literature led them to determine that greater accuracy could be obtained by making the following adjustments:
- Subtract 1.23 cm from men’s and .7 cm from women’s self-reported height.
- Add 1.86 kg to men’s self-reported weight, and subtract 1.40 kg from women’s. [A more recent study demonstrated that both men and women understate their weight, but agreed that the self-reported data could be enhanced with "calibration" algorithms, which require periodic review. It does seem likely that, with increasing attention on obesity, men may be more likely to understate their weight than they would have been just a few years ago.]
- Sensitivity and specificity of self-reported hypertension and hypercholesterolemia can be achieved by using several types of HRA questions.
(The potential to enhance validity by calibrating self-reported data offers yet another reason why — as argued in the previous post - HRA development is best left to experts and not something you should try to create on, say, SurveyMonkey.com.)
The common thread in most of the studies reviewed was that self-reported data was useful in measures of population health. (Might this foreshadow our end-goal analysis of HRAs’ overall value?) Of course, as an HRA vendor, Summex’s position obviously can be considered vulnerable to bias.
A technological assessment of HRAs, conducted by McMaster University Evidence-Based Practice Center, complicates matters further:
The validity of self-reports is always an issue. Seminal research found over 90 percent accuracy when comparing self reported cases of breast, skin, large bowel, or thyroid cancer to medical records, although accuracy was lower for self reported lung, ovary, or uterine cancer. A review of the accuracy of self reported health behaviors and risk factors in cancer and cardiovascular disease found that self reported information underestimated the proportion of persons in the general population who were actually ‘at risk’.
All tolled, these mixed results do not leave us with great confidence in self-reported health data. But they should at least plant a seed of doubt in those inclined to summarily dismiss it. Just as self-reported data perhaps cannot be assumed to be valid, nor can it be assumed to be invalid.
As mentioned at the outset, the validity of self-reported data is not the only question. We also have to ask how it compares to the alternatives — especially the much vaunted medical claims data and biometric data. In the next post, we’ll take a look at what the research shows about claims and biometric data. You may be surprised.
[This post is third in a series about health risk appraisals.]
#1 by Carol Harnett on June 12, 2012 - 10:43 am
Nice post, as always, Bob. The only thing I have to add is a brief comment from a conversation I once had with Ron Kessler from Harvard Medical School and Debra Lerner from Tufts. They both believe that people rarely misrepresent themselves with self-reported data - especially if the respondents trust where the data is being processed. That’s why - sometimes - they send data to a third party and clearly explain that in the initial materials.
#2 by Bob M. on June 12, 2012 - 12:54 pm
Thanks, Carol. I’m inferring that your use of the word “misrepresenting” (especially in the context of “trust”) implies an element of intent. But self-report also potentially can be inaccurate, or even skewed in a particular direction, for other reasons. For example, people may simply recall their health history or behaviors incorrectly. Debra Lerner clearly considered this when, in early studies of her Work Limitations Questionnaire she compared 2-week recall to 4-week recall. In fact, it’s always interesting to complete HRAs with an embedded WLQ, because you’ll answer dozens of questions about your behaviors over an unspecified time period, then you get to the very specific WLQ questions about your limitations in the last 2 weeks. Shows that Lerner is highly aware of time sensitivity and recall.
The studies I came across suggest that, even in clinical interviews, people’s recall of cholesterol levels is “systematically biased” (“self-enhancing”) several weeks after a screening, though their recall of their risk level is fairly accurate. I would not assume any intentional misrepresentation in these cases.
Even for body weight, I would venture to *guess* (a worthless endeavor, I know) that HRA participants sometimes respond with what they expect their weight to be next week or next month, which they perceive as their true weight (as opposed to the extra 20 pounds they’ve carried around the last 10 years). So it’s not necessarily an intent to misrepresent. They may believe that their technically inaccurate response is actually more precise. Our image of ourselves often conflicts with our reality, and I suspect HRAs often reflect the image.
Or, is that just me?
#3 by Rex A. Wilcox on June 12, 2012 - 2:08 pm
Bob, always enjoy your insight.
I am in favor of (where/when technology allows for it) allowing the individual to self-report their biometric data with the option (if exact measure is unknown) to select one of the following answers; Too High, Good, Too Low, Unknown. These self-reported responses are collected and then for the purpose of the final assessment/report can be overridden by measured data collected by a lab or screening vendor. Allowing for this approach provides insight into how many aren’t aware of their own numbers as well as providing baseline measurable biometric data alongside the self-reported behaviors.
About a year ago there was a lot of noise in my neck of the woods regarding the value of HRAs. Much of this noise was based upon the premise that the self-reported biometric values are highly questionable. Much of this has died down as many of those expressing the concerns moved to incorporating a screening to gather the measurable data to pair with the HRA (which is where I see the true value of the HRA)
Curious to hear additional feedback!