Totally Meta

For a poor an economically challenged liberal, I have a dirty secret. I sneak over to the WSJ about once a day.

Well, the secret isn’t that dirty: I don’t read the editorials (ick, ick, ick). Nope. I’m a fan of Carl Bialik, The Numbers Guy.

I’m more or less against fact-based science discussions. Especially when statistics are used by people who haven’t looked at the work. But Bailik has a great way of making numbers seem accessable. His discussion of the meta-analysis on the disparaged drug Avandia is a case in point.

The big news yesterday that the diabetes drug Avandia may pose cardiac risks was based on something called a meta-analysis. It’s a type of research that has some significant drawbacks, but also some unique advantages.

In a meta-analysis, researchers pool results from different studies — in this case, Cleveland Clinic cardiologist Steven Nissen and statistician Kathy Wolski analyzed 42 studies. Those studies were done by many different people, and as you might expect, there was wide variation between them. Sometimes Avandia was compared with a placebo and sometimes with alternate treatments. Adverse events — namely heart attacks shown to occur with higher frequency among Avandia users — may not have been identified consistently across the different trials. And if they weren’t, Dr. Nissen would have no way to know, because he was looking at study summaries and not patient-level data. The limitations of this “study of studies” filled a lengthy third paragraph in an accompanying New England Journal of Medicine editorial.

So why, then, use meta-analysis at all? Because for drug dangers that are rare enough, even studies of thousands of patients might not suffice to separate a real risk from random statistical variation. Combining tens of thousands of patients who underwent the treatment separately, under different protocols and supervision, may be the only way to clear thresholds for statistical significance.

He goes on to clearly describe the strengths and weaknesses of the technique; explaining the importance of the variable currently called p; when meta-analysis are useful and to explain why both sides tend to fight over the issue of whether a meta-analysis is valid.

I love statistics. (Actually, since I haven’t discussed this face to face with statistics, I should probably call it a crush, but you get the idea.)

As an example, most people, when confronted with a statistics example involving doctors, cancer patients and risk would probably change the channel. Me – I buy the book! From Joel Best’s More Damn Lies and Statistics (the sequel to Damn Lies and Statistics),

Consider the following word problem about women receiving mammograms to screen for breast cancer (the statements are, by the way, roughly accurate in regard to women in their forties who have no other symptoms):

The probability that [a woman] has breast cancer is 0.08 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram. Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer.

Confused? Don’t be ashamed. When this problem was posed to twenty-four physicians, exactly two managed to come up with the right answer. Most were wildly off: one-third answered that there was a 90 percent probability that a positive mammogram denoted actual breast cancer; and another third gave figures of 50 to 80 percent. The correct answer is about 9 percent.

Let’s look carefully at the problem. Not that breast cancer is actually rather rare (0.8 percent); that is, for every 1,000 women, 8 will have breast cancer. There is a 90 percent probability that those women will receive positive mammograms – say, 7 of the 8. That leaves 992 women who do not have breast cancer. Of this group 7 percent will also receive positive mammograms – about 69 cases of what are called false positives. Thus a total of 76 (7+69=76) women will receive positive mammograms, yet only 7 of those – about 9 percent – will actually have breast cancer. The point is that measuring risk often requires a string of calculations. Even trained professionals (such as doctors) are not used to calculating and find it easy to make mistakes. [my emphasis]

That is why fact-based science discussions fail. Not because the facts are wrong, but because any discussion of the issue won’t fit into a 30 second interview and boil down to a 25 word text snippet.

This is where framing science needs to be used. You need to be able to tell a story about how science works, how scientific uncertainty works without getting people nervous. Perhaps the fundamental difference between a scientist and a non-scientist is that the latter sees danger in uncertainty, the former sees an opportunity to write a grant proposal.

To be able to frame science, you need ideas, examples, and good stories. Like the Avandia study discussed by the Numbers Guy or some of the topics on the very entertaining Freakonomics blog by Steven Levitt and Stephen Dubner.

But sometimes – I just love the idea for itself. Statistics about statistics. Because that is just sooo totally meta.

%d bloggers like this: