After ANOVA Testing: What’s next? The Power of Fisher’s Least Significant Difference Method(LSD).
Introduction
As you may know, An ANOVA test is used when you have three or more groups of categorical data as independent variables and a continuous variable as a dependent variable. The test is an attempt to determine if there is a statistically significant difference between each group’s mean value. To explain this simply, let’s look at an example.
Let’s say we want to compare academic performance of the Top American Universities. In this case, our dependent variable could be each University’s average GPA. The independent variables (the one we want to compare) would be the Universities themselves. We may find the following results.
Before we can conduct the actual ANOVA test, We need to make a null and alternative hypothesis. In our case, the null hypothesis would be that the average GPAs would all be relatively equal. In other words, There is no statistically significant difference in average GPA between the Universities.
Using an ANOVA test, we would either reject or fail to reject the null hypothesis.This is a great first step. However, the ANOVA test does not give us any further information. If we are able to reject the null hypothesis, we have proven that there is a difference between the Universities’ average GPA but we can’t say to what extent. The ANOVA test does not tell us what Universities differ, only that there is at least one University that differs enough in average GPA to meet our threshold of significance (α).
Fisher’s Least Significant difference
But, what if we what to know which school or schools’ differ? The most simple solution is to compare the means of a pair (two school’s average GPAs). This is commonly referred to as ‘pairwise comparisons’.
Fisher’s Least Significant Difference (LSD)was the first pairwise comparison technique developed in 1935. So how does the LSD method work?
Let’s talk about this formula a little bit. First, this formula comes from the fact that assuming the null hypothesis is true (college mean GPAs are equal), then the t statistic for a pairwise comparison is equal to the first formula. Secondly, The numerator terms refer to each group’s mean value and the mean square error is the residual mean square error from an ANOVA test.
You can essentially think of the LSD method as a set of individual t tests. In fact, the only difference between these two methods is that the LSD method uses the pooled Standard Deviation (SD) from all of the groups in our dataset. While the t test method computes the pooled SD only from the two groups being compared. This difference is significant because it increases Power.
As you can see, A higher Power corresponds to a lower β. Thus, using Fisher’s LSD we lower our chances of a Type II error (false negative). In our case, this means using Fisher’s LSD we lessen our chances of not finding a significant difference between two schools’ GPAs when there is one. The one drawback to Fisher’s LSD is that in inflates Type 1 error (false positive). Which you may know intuitively, because Type 1 error and Type II error are inversely correlated.
There is one other possible advantage of Fisher’s LSD. It turns out that if the sample standard deviation for each group (of a pairwise comparison) is equal to it’s population standard deviation, then our pooled SD will be more accurate and we gain more degrees of freedom.
Weaknesses
So Fisher’s LSD is clearly a powerful tool to find a difference between our groups. However, it is it not perfect. The key takeaway here is that Fisher’s LSD is a good first step after an ANOVA test. However, if a significant difference is found, other metrics should be used to confirm the results.So, how do we use this method?
This formula is simply just derived from the previous formula. The n parameters are the size of our groups and t refers to the t distribution (Student distribution). The important thing about this particular t is that is the critical t value. This t value defines our rejection zones. So any value larger than this t means we have reached the rejection zones.
Please note that this image is for a particular case where α=.05. But α depends on your confidence level.(α=.05 is standard, used for 95% confidence level). MSW is the mean square error within the groups we are comparing. DFW is degrees of freedom within the groups we are comparing. Since Fisher’s LSD is a pairwise comparison, we only care about one pair(two schools) at a time.
Conclusions
The key idea here is that if the difference between the means of our pairwise comparison is at least as large of our calculated LSD then we can declare the result is significant.
Simply put, if the difference between two school’s average GPA is greater than the calculated Fisher’s LSD, then we know that our calculated t statistic is larger than our critical t value. Thus, we are in our rejection zone and can thus reject our null hypothesis and thereby show that there is a statistically significant difference between these two schools.
Since Fisher’s LSD method has its weaknesses (prone to Type 1 errors). Also, Fisher’s LSD method error gets worse with larger group sizes. So it’s really best used when you have three groups to compare. With larger groups, α tends to become inflated. (More Type I error)
A modified LSD called MSLD (M for modified) was proposed by Hayter to compensate for these issues. It is often called the Fisher-Hayter Procedure.
Using this formula is near identical to using the original Fisher’s LSD method. There are other methods as well such as Tukey-Kramer test but the Fisher-Hayter procedure results in higher power.
Please remember that these tools are used for pairwise comparisons, so the procedure would be repeated for each pair that needs to be compared.
Key Takeaways
Fisher’s LSD method is not perfect. It is really best used when you have a small amount of groups and since it relies on pairwise comparisons, it can also be very time consuming for multiple comparisons. However, due to it being very statistically powerful (Type II error is lower),it is certainly useful in the right cases. Lastly, for larger group comparisons, the Fisher-Hayter Procedure can be very useful as it retains the powerfulness from the original Fisher method.