Big Data, Small Data, Any Data At All

Why is it that I still remember that the formula for the volume of a sphere is 4/3π r³, which I learned in tenth grade geometry? And why is it that I never even heard of a p-value, the measure commonly used to assess whether a result is “statistically significant,” until I was in medical school? I haven’t had any occasion to compute the volume of a sphere since I took calculus in college, but I have to interpret statistical findings all the time. Something is not right here.

Understanding at least the rudiments of statistics matters—and not just to me, a physician who has to make decisions about how to treat patients by evaluating articles in the medical literature that rely on statistical methodology. Understanding basic statistics matters to everyone. You need to know some statistics to realize that it is more accurate to measure the population by using sampling techniques than by trying to count everyone. You need to know some statistics to understand why Nate Silver, with his FiveThirtyEight website, was so much more on target in his predictions about the 2012 presidential elections than anyone else. And you need to know some statistics to decide, as a patient, how to evaluate the options your physician presents you with.

Just this morning, I read an article in the first section of the NY Times “Study Discounts Testosterone-Suppressing Therapy for Early Prostate Cancer.” It turns out that millions of men with early stage prostate cancer, mainly men over the age of 65, have been treated with “Androgen Deprivation Therapy” (ADT), either by bilateral orchiectomy (surgical removal of the testes) or by drugs. A new study, published in JAMA Internal Medicine, concludes that ADT in such men does not prolong life. It does cause lots of side effects, ranging from osteoporosis to weight gain, to decreased libido, to diabetes. The article quotes one expert who was not involved in the study as saying that the findings were “eye-opening and even alarming.” According to editorial writers from the Dana Farber Cancer Institute, the treatment is a good candidate for inclusion in the “Choosing Wisely” campaign, a national effort to eliminate the use of “low value medicine;” that is, treatments that achieve little, given their cost. The article fits in nicely with a major theme of JAMA Internal Medicine, which has a section called “Less is More.” It’s a theme that resonates with me as well: I often argue on this blog that certain treatments, especially when provided to frail, older individuals, may cause more harm than good. Finding that a commonly used treatment, such as ADT in older men, doesn’t do what it promises, would not be at all surprising to me. But is it true?

I looked up the article, which isn’t actually in the print issue of the journal yet; it was published in the “online first” section, which gets important articles distributed quickly. The authors looked at data on 66,717 men age 66 or older with localized prostate cancer diagnosed between 1992 and 2009. They defined “primary ADT” as orchiectomy or the use of a drug such as a luteinizing hormone releasing agonist (a drug that stimulates the pituitary to signal the testes to make testosterone until they run out, at which point testosterone levels fall) as the sole cancer therapy given to men with localized prostate cancer within 6 months of diagnosis. The outcomes they were interested in were cancer specific mortality (that is, the death rate from prostate cancer) and overall mortality. So far so good. 

But since this was not a randomized study in which some men got ADT and others received conservative management (ie no treatment unless symptoms develop), with the selection made based on the flip of a coin, there was no reason to believe that the two groups of men would be similar to one another. In fact, they were quite different. The men who got ADT were a good bit older than those who did not (average age 79 vs 77). They were considerably sicker, with higher rates of other diseases such as heart disease or lung disease. And they were far more likely to have “high risk” prostate cancer, based on the characteristics of the cells in their tumors (47.7% vs 23%). Their PSA scores were also much higher (an average of 19.5 in the ADT group compared to 11.1 in the other men, where 4 is the typical cutoff for normal). Simply comparing the outcomes in these 2 very dissimilar groups of men would not tell the whole story. Somehow, the authors needed to try to compensate for the inherent differences between the men. The only way to do that (other than scrapping this approach entirely and randomizing men to get ADT or some other treatment), is to build a statistical model.

Build a model the study authors did. The specifics of what they actually did are too complicated to describe here. I’m not sure I fully understand what they did, but it involved a technique called “Instrumental Variable Analysis,” known as IV. Suffice it to say that when they used this approach to try to adjust for all the differences between the groups (only some of which they could specify), they concluded that the 15-year prostate cancer specific survival rate was 85.4% in both groups. And when they used a different method, the Cox multivariate model, they found the mortality rate was 2.4/100 in the ADT group and 1.1/100 in the group treated with conservative management or, after attempting to adjust for differences based on what was known about other illnesses, PSA levels, etcetera, the group treated with ADT was 1.53 times more likely to die.

What the reader needs to understand is that the results of the study depend entirely on which model you choose. If you select IV, and the authors try hard to make the case that this is an excellent choice, but which some experts think is a flawed approach, you find that ADT and conservative therapy are equivalent. If you select the more conventional approach, you find that ADT is actually worse than watchful weighting. Since neither model predicts that ADT is better than conservative management, perhaps it follows that ADT is just a bad choice for the treatment of early prostate cancer in older men. The right conclusion, I think, is that we don’t actually know what to make of ADT. If we chose yet another model, perhaps we would find that ADT is superior.

Learning about different study designs—which ones you can trust, which ones are merely suggestive and which have to be confirmed using a better, more reliable approach—is what kids should be learning in high school and college. Learning about probability and statistics is what kids should be learning, not trigonometry and solid geometry. Our math curriculum reflects seventeenth century mathematical knowledge (it typically includes elementary algebra, Euclidean geometry, and perhaps calculus, created in the fourth century BCE and the seventeenth centuries respectively). 

Today, big data is all the rage and there is a growing enthusiasm for learning how to milk large data sets for useful information. But the reality is that it’s not just big data that’s important and it’s not just important for a small cadre of people. We all need to learn how to make sense of what we read in the newspapers, of what our doctors tell us about different treatments. And to do that, we need to develop basic statistical literacy.

Words, Words, Words

Physicians are not known for their communication skills. Despite sessions in medical school and during residency addressing topics such as “breaking bad news” and “discussing prognosis,” clinicians still do not perform well. A new program for both doctors-in-training and nurse practitioner students sought to improve the poor track record. 

Participating students attended eight 4-hour workshop sessions addressing communication in end-of-life care. Each session included a short lecture, a demonstration by a faculty member of good communication, a practice session for the trainees using a simulated patient, and a discussion. A total of 184 students completed the workshops; another 222 completed “usual education.” (The vast majority were physicians- in-training and only a few were NP students, so I will restrict my comments to doctors). The students were evaluated for their proficiency in communication and their skill in providing end-of-life care after the educational intervention, with physicians, patients, and family members all contributing to the evaluation process. The bottom line: the workshops did not appear to accomplish anything. Students who participated in the workshops performed at almost exactly the same level as those who received the “usual education,” both in their communication skills and in their delivery of end-of-life care. In fact, the only measurable difference between the two groups was that patients whose physician had taken the training were more likely to end up being depressed than those whose physician did not. What are we to make of these findings?

The authors offer several interesting possible interpretations. They point out that an earlier study showed that patients who understood they had a very poor prognosis were more likely to rate their physicians’ communication skills as poor than those who incorrectly believed their prognosis was pretty good. So it is entirely possible that what palliative care physicians mean by “good communication skills” is not what patients and families mean. Palliative care doctors think that good communicators give patients a realistic understanding of their clinical situation and elicit their patients’ preferences for future medical care, all in a compassionate and caring way. Maybe patients and families equate “good communication” with encouragement, or instilling hope, or holding out the prospect of cure, however implausible cure may be. Maybe objective assessments by trained faculty are a better way of evaluating success than are patient and family reports; it turns out that when faculty did the evaluating, they found that students did improve after the workshop. Maybe expecting that the students would do a better job providing end-of- life care after a workshop on communication was naïve; after all, excellent end-of-life care includes expert pain management, good diagnostic skills, and appropriate referral to other clinicians, not just good communication. 

I’d like to suggest a different conclusion. Perhaps it’s time to stop investing so much effort in trying to change physicians. We should turn instead to a radically different way of educating patients and helping them make the difficult decisions they face near the end of life. One of my colleagues has taken just such a tack. He designs short videos to show patients and their families the medical conditions they have and the interventions they might be offered. Multiple studies have now demonstrated that patients who watch these videos have a much clearer idea of what is at stake and express different preferences from patients who hear a verbal description of their disease and the options for treatment. The videos do not replace clinicians; rather, they give patients a strong foundation on which to build when they talk with their physician. They allow doctors to go beyond explaining the basics and they allow patients to apply the information they have learned to their specific situation.

Most of my career has been devoted to trying to be the best possible communicator with my patients, on the one hand, and to writing books and articles to help clinicians and patients make better decisions, on the other. So it pains me to think that this intensely verbal approach may just not be as effective as carefully constructed videos that show the realities of advanced illness and contemporary treatment. I will continue to write (this blog included) because that’s what I do. But I will also partner with my young colleague to create scientifically accurate videos, reviewed by experts, that complement all those words.

Perhaps Eliza Doolittle put it best, in My Fair Lady:

Words, words, words
I’m so sick of words…

Sing me no song, read me no rhyme, 
Don’t waste my time, show me!
Please don’t implore, beg or beseech,
Don’t make a speech, show me!