Type I error in multiple comparison tests in analysis of variance
Abstract
In a hypothesis test, a researcher initially fixes a type I error rate, that is, the probability of rejecting the null hypothesis given that it is true. In the case of means tests, it is important to present a type I error that is equal to the nominal pre-fixed level, such that this error remains unchanged across various scenarios, including the number of treatments, number of repetitions, and coefficient of variation. The purpose of this study is to analyse and compare the following multiple comparison tests for the control of both conditional and unconditional type I error rates, depending on a significant F-test in the analysis of variance: Tukey, Duncan, Fisher’s least significant difference, Student–Newman–Keuls (SNK), and Scheffé. As an application, we present a motivation study and develop a simulation study using the Monte Carlo method for a total of 64 scenarios. In each simulated scenario, we estimate the comparison-wise and experiment-wise error rates, conditional and unconditional on a significant result of the overall F-test of analysis of variance for each of the five multiple comparison tests evaluated. The results indicate that the application of the means tests based only on the significance of the F-test should be considered when determining the error rates, as this can change them. In addition, we find that Fisher’s test controls for the comparison-wise error rate, the Tukey and SNK tests control for the experiment-wise error rate, and the Duncan and Fisher tests control for the conditional experiment-wise error rate. Scheffé’s test does not control for any of the error rates considered.
Downloads
References
Bernhardson, C. S. (1975). Type I error rates when multiple comparison procedures follow a significant F test of ANOVA. Biometrics, 31(1), 229-232. DOI: https://doi.org/10.2307/2529724
Biase, N. G., & Ferreira, D. F. (2011). Testes de igualdade e de comparações múltiplas para várias proporções binomiais independentes. Revista Brasileira de Biometria, 29(4), 549-570.
Boardman, T. J., & Moffitt, D. R. (1971). Graphical Monte Carlo Type I error rates for multiple comparison procedures. Biometrics, 27(3), 738-744. DOI: https://doi.org/10.2307/2528613
Cardellino, R. A., & Siewerdt, F. (1992). Use and misuse of statistical tests for comparison of means. Revista da Sociedade Brasileira de Zootecnia, 21(6), 985-995.
Girardi, L. H., Cargnelutti Filho, A., & Storck, L. (2009). Erro tipo I e poder de cinco testes de comparação múltipla de médias. Revista Brasileira de Biometria, 27(1), 23-36.
Gonçalves, B. O., Ramos, P. S., & Avelar, F. G. (2015). Test Student-Newman-Keuls bootstrap: proposal, evaluation, and application productivity data of soursop. Revista Brasileira de Biometria, 33(4), 445-470.
Henrique, F. H., & Laca-Buendía, J. P. (2010). Comportamento morfológico e agronômico de genótipos de algodoeiro no município de Uberaba - MG. FAZU em Revista, 7, 32-36.
Perecin, D., & Barbosa, J. C. (1988). Uma avaliação de seis procedimentos para comparações múltiplas. Revista de Matemática e Estatística, 6, 95-103.
R Core Team (2020). R: A language and environment for statistical computing. Vienna, AT: R Foundation for Statistical Computing. Retrieved on Jan. 9, 2021 from URL https://www.R-project.org/
Ramos, P. S., & Vieira, M. T. (2014). Bootstrap multiple comparison procedure based on the F distribution. Revista Brasileira de Biometria, 31(4), 529-546.
Rodrigues, J., Piedade, S. M. S., & Lara, I. A. R. (2016). Aplicação condicional de testes de comparação de médias a um resultado significativo do teste F global na análise de variância. Revista Brasileira de Biometria, 34(1), 1-22.
Saville, D. J. (2014). Multiple comparison procedures - Cutting the Gordian knot. Agronomy Journal, 107(2), 730-735. DOI: https://doi.org/10.2134/agronj2012.0394
Souza, C. A., Lira Junior, M. A., & Ferreira, R. L. C. (2012). Avaliação de testes estatísticos de comparações múltiplas de médias. Revista Ceres, 59(3), 350-354. DOI: https://doi.org/10.1590/S0034-737X2012000300008
DECLARATION OF ORIGINALITY AND COPYRIGHTS
I Declare that current article is original and has not been submitted for publication, in part or in whole, to any other national or international journal.
The copyrights belong exclusively to the authors. Published content is licensed under Creative Commons Attribution 4.0 (CC BY 4.0) guidelines, which allows sharing (copy and distribution of the material in any medium or format) and adaptation (remix, transform, and build upon the material) for any purpose, even commercially, under the terms of attribution.
