Rejecting (or failing to) the null hypothesis

[Seiryuu]

My group had recently done a choice experiment on Artemia with respect to preference of light intensity (three zones: one covered with black plastic, one covered with a fabric that's weaved so some light gets through, and one with no barrier). We performed two trials and the data we collected allowed us to both reject and fail to reject the null hypothesis using the chi-square test and the table of critical chi-square values (separately). So does that mean that there is some significance? No significance?

[ascendancy]

How did you set up the chi square values. With what input?

[Seiryuu]

From the lab manual I have, the formula was [math]\chi^2=\frac{(observed-expected)^2}{expected}[/math]. I know that the expected number is the number of replicates we tested divided by the number of possible choices (i.e., 3). In my group's first trial, the replicates preferred to stay in the darker region such that our calculated chi-square value was greater than the critical chi-square of 2 degrees of freedom (5.99), whereas in our second trial, the replicates showed no significant preference to any of our choices and as such the calculated chi-square value of trial 2 was less than the critical chi-square value of 5.99.

 

In other words:

• We were able to reject the null hypothesis in trial 1.
• We were unable to reject the null hypothesis in trial 2.

[CharonY]

Assuming that all trials were done identically, one would use the different trials essentially as coming from the same sample population. In other words, you would pool the data and make one single test with all data combined.

[Seiryuu]

Assuming that all trials were done identically, one would use the different trials essentially as coming from the same sample population. In other words, you would pool the data and make one single test with all data combined.

 

So if the trials were performed identically in every way, I would just take all the data and make one "big trial?"

So I'm inferring that if there was something slightly different, they'd have to stay separate and my all overall conclusion would be that they're somewhat significant?

[CharonY]

As a rule of thumb, if you repeat a given experiment or trial (i.e. all parameters are the same) and end up with different results for each data set, the chances are high that your sample size was too small. A larger set (by pooling) is required to demonstrate that whether there was an effect or not.

If the experiment was set up (deliberately) different, they have to be treated as coming from different sample populations. However, if you suspect something was off, but do not know what, you still would have to pool the data in the hope that whatever bias you may have introduced gets drowned out, since you cannot infer which of the two trials reflects the true population best. If you just took one of these results, you would just be guessing and that defies the whole purpose of doing statistics.

 

In short, if both experiments are conducted the same way, treat it as one sample population. Even if you suspect something is off, but don't know why treat them as one. If you know that there are technical differences (e.g. older animals, different test parameters etc.) you could do them differently and say e.g. under condition 1 we found no differences, however by altering the experimental parameter X to Y we found a significant difference in whatever.

 

The important bit is that even you expect something to happen but do not see it, do not try to fix your data or statistical analysis in such a way that it fits your expectation. Again, that would be bad science and defeats the whole purpose of experimentation and statistics.

[Seiryuu]

I am really, really grateful for your expertise, Charon. Thank you very much!