Archive for May 23rd, 2007|Daily archive page

Totally Meta

For a poor an economically challenged liberal, I have a dirty secret. I sneak over to the WSJ about once a day.

Well, the secret isn’t that dirty: I don’t read the editorials (ick, ick, ick). Nope. I’m a fan of Carl Bialik, The Numbers Guy.

I’m more or less against fact-based science discussions. Especially when statistics are used by people who haven’t looked at the work. But Bailik has a great way of making numbers seem accessable. His discussion of the meta-analysis on the disparaged drug Avandia is a case in point.

The big news yesterday that the diabetes drug Avandia may pose cardiac risks was based on something called a meta-analysis. It’s a type of research that has some significant drawbacks, but also some unique advantages.

In a meta-analysis, researchers pool results from different studies — in this case, Cleveland Clinic cardiologist Steven Nissen and statistician Kathy Wolski analyzed 42 studies. Those studies were done by many different people, and as you might expect, there was wide variation between them. Sometimes Avandia was compared with a placebo and sometimes with alternate treatments. Adverse events — namely heart attacks shown to occur with higher frequency among Avandia users — may not have been identified consistently across the different trials. And if they weren’t, Dr. Nissen would have no way to know, because he was looking at study summaries and not patient-level data. The limitations of this “study of studies” filled a lengthy third paragraph in an accompanying New England Journal of Medicine editorial.

So why, then, use meta-analysis at all? Because for drug dangers that are rare enough, even studies of thousands of patients might not suffice to separate a real risk from random statistical variation. Combining tens of thousands of patients who underwent the treatment separately, under different protocols and supervision, may be the only way to clear thresholds for statistical significance.

He goes on to clearly describe the strengths and weaknesses of the technique; explaining the importance of the variable currently called p; when meta-analysis are useful and to explain why both sides tend to fight over the issue of whether a meta-analysis is valid.

I love statistics. (Actually, since I haven’t discussed this face to face with statistics, I should probably call it a crush, but you get the idea.)

As an example, most people, when confronted with a statistics example involving doctors, cancer patients and risk would probably change the channel. Me – I buy the book! From Joel Best’s More Damn Lies and Statistics (the sequel to Damn Lies and Statistics),

Consider the following word problem about women receiving mammograms to screen for breast cancer (the statements are, by the way, roughly accurate in regard to women in their forties who have no other symptoms):

The probability that [a woman] has breast cancer is 0.08 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram. Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer.

Confused? Don’t be ashamed. When this problem was posed to twenty-four physicians, exactly two managed to come up with the right answer. Most were wildly off: one-third answered that there was a 90 percent probability that a positive mammogram denoted actual breast cancer; and another third gave figures of 50 to 80 percent. The correct answer is about 9 percent.

Let’s look carefully at the problem. Not that breast cancer is actually rather rare (0.8 percent); that is, for every 1,000 women, 8 will have breast cancer. There is a 90 percent probability that those women will receive positive mammograms – say, 7 of the 8. That leaves 992 women who do not have breast cancer. Of this group 7 percent will also receive positive mammograms – about 69 cases of what are called false positives. Thus a total of 76 (7+69=76) women will receive positive mammograms, yet only 7 of those – about 9 percent – will actually have breast cancer. The point is that measuring risk often requires a string of calculations. Even trained professionals (such as doctors) are not used to calculating and find it easy to make mistakes. [my emphasis]

That is why fact-based science discussions fail. Not because the facts are wrong, but because any discussion of the issue won’t fit into a 30 second interview and boil down to a 25 word text snippet.

This is where framing science needs to be used. You need to be able to tell a story about how science works, how scientific uncertainty works without getting people nervous. Perhaps the fundamental difference between a scientist and a non-scientist is that the latter sees danger in uncertainty, the former sees an opportunity to write a grant proposal.

To be able to frame science, you need ideas, examples, and good stories. Like the Avandia study discussed by the Numbers Guy or some of the topics on the very entertaining Freakonomics blog by Steven Levitt and Stephen Dubner.

But sometimes – I just love the idea for itself. Statistics about statistics. Because that is just sooo totally meta.

Advertisements

When Less Is Moore

I am not a big Michael Moore fan.

Don’t get me wrong; I understand what he’s trying to do – balance right wing demagoguery with hard-core, left-wing propaganda. Most of his documentaries leave me sputtering something along the lines of ”But, but, but… of course they did that! They’re scum. That’s obvious.”

But even when Moore starts burning the Bush, I still can’t seem to start waving the flag.

That’s why my ears perked up when a report was aired on German radio about two Canadian filmmakers, Rick Caine and Debbie Melnyk, setting out to follow in Michael Moore’s footsteps. No, not just the documentary-making ones, the physical ones.

Caine and Melnyk, self admitted Canadian lefties, started out to make a documentary about their hero, Michael Moore. It didn’t go well. John Anderson at the International Herald (or AP?) interviewed the husband and wife team,

“What he’s done for documentaries is amazing,” said Melnyk, 48, a native of Toronto and a freelance TV producer, who even now expounds on the good she says Moore has done. “People go to see documentaries now and, as documentary makers, we’re grateful.”

But according to Caine, 46, an Ohio-born journalist and cameraman, the freewheeling persona cultivated by Moore, and the free-thinking rhetoric expounded by his friends and associates were not quite what they encountered when they decided to examine his work. “As investigative documentarists we always thought we could look at anything we wanted,” Caine said. “But when we turned the cameras on one of the leading figures in our own industry, the people we wanted to talk to were like: ‘What are you doing? Why are you throwing stones at the parade leader?'”

Melnyk added, “We were very lonely.”

The movie Manufacturing Dissent was born.

Moore’s missteps included faked footage, creative editing (who’d a thought?!) and perhaps most damning, leaving out the interview that was the premise for Moore’s first big hit, Roger & Me. Apparently Moore did actually get an interview with GM CEO Roger Smith, he just didn’t include the footage in the movie. The Canadian filmmakers swept the cutting room floor and included the ‘lost’ scenes in their documentary. Ouch.

They tried to get an interview or at least a little camera time with the ‘giant’ of documentary filmmaking. But even after staging several Moore-ian stunts, Caine and Melnyk’s efforts were apparently less successful than the supersized, lefty superhero.

Having lefties attack Michael Moore (well, having anyone attack Michal Moore) is a highlight in any Fox “News” cycle. But the filmmakers, after seeing the distortion a few nips and tucks their hero could produce, were marginally sceptical about appearing on an edited Fox broadcast. They did however agree to appear live. The result was – well – interesting.

Even more interesting is the resultant comment produced after they appeared with Martha MacCallum on Fox’s “The Live Desk.”

“We said: ‘This is crap. We do not want to become poster kids for the right-wing media. No, we haven’t seen the light and converted.’ That is exactly what they were thinking,” Melnyk says. “But we were intent on telling them that it’s not only Michael Moore who is lying and cheating, it’s mainstream news organizations and George Bush.”

Adds Caine with a laugh: “I could hear a person in New York screaming into my earpiece: ‘Get that asshole off the air.’ They cut us off.”

The couple, in short, refused to bitterly attack Moore, even though his handlers once had them kicked out of the audience at one of his speeches.

Sicko

Now Moore is headlining again with his newest effort Sicko.

The documentary premiered on May 20 in Cannes to fairly good reviews. According to the Guardian,

Michael Moore’s Sicko, which received its first-ever screening in front of a packed, early-morning audience in Cannes yesterday, is a far more thoughtful and measured piece of film making than his Palme d’Or winning rant, Fahrenheit 9/11It is, however, unlikely to repeat the commercial success and global notoriety of its predecessor simply because its concerns are more parochial, focusing on the American health service and the system’s iniquities compared with those of Cuba, Canada, France and the UK.

The film is a campaigning attack on the profit-driven US healthcare system which, argues Moore, is weighted in favour of the drugs and insurance companies rather than the patients. He begins by saying that 50 million Americans can’t afford health cover and goes on to state that many of the 250 million who do pay insurance are not as well-protected as they might think.

Always good for a controversy or two, Moore filmed  a brief segment, in Cuba with 9/11 responders. This was to highlight the differences in healthcare between the ‘richest nation in the world’ and – well – Cuba.

Unfortunately, or perhaps predictably, this got the US Treasury Department in a huff; Moore might not have gotten the appropriate licenses. (It should be noted that this would have even irritated the USTD under Clinton who famously tightened the embargo in 1996.) One effect the Moore visit to Cuba will likely have is to make the exile Cubans in Florida grumpy. Probably not a good way to get more liberal/democratic support so close to an important presidential election in that swing state. But hey; let’s let embargos be embargos, shall we?

It remains to be seen what effect this movie will have. I’m sure the British will smirk when watching Moore wax poetic on their much maligned National Health System since the NHS is usually described in the British press as one of the roots of all evil. As a matter of fact, some have already started smirking,

Our own dear National Health Service also comes in for lavish praise. There’s a particularly comic sequence in which Moore marches round Hammersmith hospital in London searching for the payments section. Eventually he finds the cashier’s office: much mock incredulity ensues when he discovers its purpose is not to receive money from patients, but to pay out cash to those of them who cannot afford their travel expenses.

I guess I could hope though. I’d like to see Michael Moore’s next effort be about the food industry; a major campaign to get people in America to lose weight.

Then we might finally get what I’d really enjoy – less of Moore. That would be a documentary I’d go watch.