Friday, December 19, 2008

Why I Never Trust Studies (Part II)

In the previous posting I took quite the cynical look at how the results from studies can be manipulated by researchers or journalists. But in the previous post I did not question the results themselves, simply the methodology taken, so I left open the possibility that with better methodology one might obtain the correct result. Unfortunately, even with proper methodology an ethical researcher might still obtain an incorrect result.

Moreover, groups of independent researchers may collectively obtain inaccurate results. A professor once related to me the story about a series of groups of astrophysics researchers who were observing a certain astronomical phenomenon. I won’t go into details but after extensive observations each of these groups obtained certain data, and together the data collected indicated that a particular process was occurring. With this evidence the researchers obtained millions of dollars in grants to construct a more advanced observation device to observe the process. Once it was constructed and turned on the data this more accurate device obtained indicated that all of the groups were wrong.

How could this be possible? Some of the researchers hypothesized that while they were constructing the observation device that the process ‘turned off’. Maybe. Or maybe the way they obtained the data unintentionally biased their results since they were looking to obtain a certain result (this is one reason that double blind studies are done).

But again, this is a methodology problem right? If they changed the way they handle data then it would remove this problem right?

Well not necessarily. Lets take a look at an example. Suppose that a medical researcher wants to test a drug on a set of 100 human patients. For simplicity, say that the ailment has a 50% chance of killing any individual patient over a set time frame. Someone might think that if the drug works then more than 50 of the patients will live, if it doesn’t then only 50 will survive. But compare this to flipping a coin 100 times, will you always get 50 heads and 50 tails? Of course not.

This is where researchers introduce what is called ‘confidence levels’. In other words the number of surviving patients must be so large that the researcher feels confident that the drug must be working. On our given data, the 90% confidence level is around 56 surviving patients. But suppose for a second that the drug doesn’t work at all and the patients still die at a 50% rate, how often then would the researcher obtain that 90% confidence level? How often would they reach the 85% confidence level (around 55 patients)? Creating a simple program to randomly generate patients who die at a 50% rate shows that the researcher would publish erroneous results at the 90% confidence level ~7% of the time. At the 85% confidence level their results would be wrong ~10% of the time.

One might say that this is not significant, that the probability is still fairly small. And this is true, but it is important to remember that there is still a significant probability that the results are incorrect.

Confidence levels vary between different fields, some fields require a greater threshold and some fields require less. So consider yourself warned, next time you read about a study, check to see what the confidence level and sample size were before you consider whether or not to believe it.

No comments:

Post a Comment