Multiple Testing Question/Confusion

[stackcanary]

Hi, I've been looking at material all day and am still confused about multiple testing. For the assignment, we're given a paper where three groups (2 experimental, 1 control), are put on controlled diets, have have biomarkers and memory assessed after 3 months. The question is why it was appropriate for the authors to no correct for multiple testing. They performed post-hoc t-tests for differences before and after the intervention.

 

Most of the documents I've been able to find say that multiple testing correction is used when you're comparing a massive number of metrics and need to make your significance more stringent to keep out false positive results, but those are mostly in relation to gene studies. If there are about 18 metrics being compared between 3 groups (or, each of the two experimental groups to the control group), would they not correct because the number of metrics is relatively low, or would they not compare because the number of groups is low? I'm pretty confused and would love some guidance on this.

 

Thanks!

[CharonY]

Basically you are correct. If several parameters are tested it it would be more appropriate to adjust for multiple hypothesis testing. Since the number of tests factors in exponentially, it is usually not appropriate to neglect it. In practice it depends a little bit on whether they really did multiple hypothesis testing. Something that often done is ANOVA f-test followed by the mentioned post-hoc t-test. If the latter only includes two samples (e.g. the groups with the biggest means difference), it does not require adjusting, of course. However, if every combination is tested, adjustments are necessary. A number of post-hocs do control family-wise errors, though. In the end, it depends a little how the authors explain and interpret the data.

 

It should be noted that there are quite some papers around that fail to account for multiple hypothesis testing, though it depends a bit on the field, whether they caught up to it. Especially in molecular biomarker papers many still fail to do it.

Edited March 16, 2011 by CharonY