Thursday, December 11, 2008

Why I Never Trust Studies (Part I)

Every so often you'll hear a study come out that declares something like 'study shows health benefits to doing X' or 'study proves that Y is true!'.

Now, reporters like these sort of declarations because they are definitive and something that might get them on the front pages. Researchers like them too because its a lot easier to get funding after you've published some definitive results. You'll never see a front page headline stating 'study on health benefits of Z inconclusive'. And you can't really publish a decent article with those results either, but scientifically sometimes a null result is just as important as a positive result.

Moreover, unless the reporter has a lot of time on their hands and access to the methodology of the study you won't see the reporter digging through the methodology of the study making sure everything checks out. Generally, they let other researchers do that later. But if something is wrong in the methodology and it is found out later the correction usually doesn't front page news.

Take for example, the studies that showed that moderate alcohol drinking is good for your heart. I'm sure you've all seen articles like this or this that discuss studies that claim that drinking alcohol (especially wine) is good for you. One article you've probably not seen is this one, which points out that many studies make a fundamental flaw in their methodology and hence are questionable at best.

If you begin digging deep into the actual methodology of a study you'll find that sometimes the headlines aren't entirely true. Take for example the MSNBC article I linked to above, which outlines how drinking wine is good for your health. The chemicals that they credit for being so helpful, the resveratrol and flavanoids, are found in the grapes themselves and do not arise because of the fermentation process at all! These chemicals are also found in green teas. But an article saying 'Eating grapes good for your health' wouldn't make the front pages because everyone already knows that. Saying 'Drinking wine is good for your health' is much better.

So, okay you might think, some of the studies may use poor methodology and the results of some studies may be skewed intentionally by media and/or unethical researchers. But surely, an ethical researcher using proper methodology and using proper statistics would never publish a result that is incorrect. Right? Wrong again, and I'll prove it to you in Part 2.

No comments:

Post a Comment