Algoritmo de um teste adaptativo informatizado com base na teoria da resposta ao item para a estimação da usabilidade de sites de e-commerce
Algorithm of computerized adaptive testing to estimate the usability of e-commerce sites
Moreira Junior, Fernando de Jesus; Tezza, Rafael; Andrade, Dalton Francisco de; Bornia, Antonio Cezar
http://dx.doi.org/10.1590/S0103-65132012005000095
Prod, vol.23, n3, p.525-536, 2013
Resumo
O presente artigo propõe um algoritmo de um teste adaptativo informatizado baseado na teoria da resposta ao item, desenvolvido para estimar o grau de usabilidade de sites de e-commerce. Cinco algoritmos baseados no critério da máxima informação foram desenvolvidos e testados via simulação. O algoritmo com o melhor desempenho foi aplicado nos dados reais de 361 sites de e-commerce. Os resultados mostraram que o algoritmo desenvolvido consegue obter uma boa estimativa para o grau de usabilidade de sites de e-commerce com a aplicação de 13 itens.
Palavras-chave
Teste adaptativo informatizado. Teoria da resposta ao item. Usabilidade. Sites de e-commerce
Abstract
This paper proposes an algorithm of a computerized adaptive testing based on Item Response Theory, designed to estimate the degree of usability of e-commerce sites. Five algorithms were tested by simulation. The algorithm with the best performance was applied to real data from 361 e-commerce sites. The results showed that the algorithm could obtain good estimates for the degree of usability of e-commerce sites with the application of 13 items.
Keywords
Computerized adaptive testing. Item response theory. Usability. e-commerce sites.
References
AGARWAL, R.; VENKATESH, V. Assessing a Firms Web Presence: A Heuristic Evaluation Procedure for the Measurement of Usability. Information Systems Research, v. 13, n. 2, p. 168-186, June 2002. http://dx.doi.org/10.1287/isre.13.2.168.84
ANDRADE, D. F.; TAVARES, H. R.; VALLE, R. C. Teoria da resposta ao item: conceitos e aplicações. São Paulo: Associação Brasileira de Estatística - ABE, 2000.
DE AYALA, R. J. The Theory and Practice of Item Response Theory. New York: The Guilford Press, Wiley, 2009.
BIRNBAUM, A. Some Latent Trait Models and Their Use in Infering an Examinees Ability. In: LORD, F. M.; NOVICK, M. R. Statistical Theories of Mental Test Scores. Reading: Addison-Wesley, 1968.
CHEVALIER, A.; BONNARDEL, N. Articulation of web site design constraints: Effects of the task and designers expertise. Computers in Human Behavior, v. 23, n. 5, p. 2455-2472, 2007. http://dx.doi.org/10.1016/j.chb.2006.04.001
CHOE, P. et al. Evaluating and improving a self-help technical support Web site: Use of focus group interviews. International Journal of Human-Computer Interaction, v. 21, n. 3, p. 333-354, 2006. http://dx.doi.org/10.1207/s15327590ijhc2103_4
COHEN, J. A coefficient of agreement for nominal scales. Educacional and Psychological Measurement, v. 20, n. 1, p. 37-46, 1960. http://dx.doi.org/10.1177/001316446002000104
CROMBACH, L. J.; GLESER, G. C. Psychological Tests and Personal Decisions. Urbana: University of Illinois Press, 1957.
CYBIS, W. Ergonomia e Usabilidade: conhecimentos, métodos e aplicações. São Paulo: Novatec Editora, 2007.
EMBRETSON, S.; REISE, S. P. Item Response Theory for Psychologists. New Jersey: Lawrence Erlbaum Associates, Inc. Publishers, 2000.
FANG, X.; HOLSAPPLE, C. W. An empirical study of web site navigation structures impacts on web site usability. Decision Support Systems, v. 43, n. 2, p. 476-491, 2007. http://dx.doi.org/10.1016/j.dss.2006.11.004
FAYERS, P. M.; MACHIN, D. Quality of Life: The Assessment, Analysis and Interpretation of Patient-reported Outcomes. 2nd ed. Wiley, 2007.
FETZER, M. et al. Computer Adaptive Testing (CAT) in an Employment Context. Roswell: PreVisor, 2008. White paper.
HAMBLETON, R. K. Emergence of Item Response Modeling in Instrument Development and Data Analysis. Medical Care, v. 38, n. 9, p. 60-65, 2000. Supplement II.
HAMBLETON, R. K.; SWAMINATHAN, H.; ROGERS, H. J. Fundamentals of item response theory. Newbury Park: Sage, 1991.
HERRANDO, S. Tests adaptativos computerizados: una sencilla solución al problema de la estimación con puntuaciones perfectas y cero. In: CONFERENCIA ESPAÑOLA DE BIOMETRÍA, 2., 1989, Segovia, Espanha. Anales Segovia: Biometric Society, 1989.
IVORY, M. Y.; MEGRAW, R. Evolution of web site design patterns. ACM Transactions on Information Systems, v. 23, n. 4, p. 463-497, 2005. http://dx.doi.org/10.1145/1095872.1095876
KIERAS, D. E.; POLSON, P. G. An approach to the formal analysis of user complexity. International Journal of Human-Computer Studies, v. 51, n. 2 p. 405-434, 1999. http://dx.doi.org/10.1006/ijhc.1983.0317
LARGE, A. et al. Web Portal Design Guidelines as Identified by Children through the Processes of Design and Evaluation. Proceedings of the American Society for Information Science and Technology, v. 43, n. 1, p. 1-23, 2006. http://dx.doi.org/10.1002/meet.1450430120
LAZAR, J.; MEISELWITZ, G.; NORCIO, A., A taxonomy of novice user perception of error on the web. Universal Access in the Information Society Journal, v. 3, n. 3-4, p. 202-208, 2004. http://dx.doi.org/10.1007/s10209‑004-0095-9
LORD, F. M. A broad-range tailored test of verbal ability. Applied Psychological Measurement, v. 1, n. 1, p. 95-100, 1977. http://dx.doi.org/10.1177/014662167700100115
MAGOUTAS, B. et al. An adaptive e-questionnaire for measuring user perceived portal quality. International Journal of Human-Computer Studies, v. 68, n. 10, p. 729-745, 2010. http://dx.doi.org/10.1016/j.ijhcs.2010.06.003
MOREIRA JUNIOR, F. J. Aplicações da Teoria da Resposta ao Item (TRI) no Brasil. Revista Brasileira de Biometria, v. 28, n. 4, p. 137-170, 2010.
MUÑIZ, J.; HAMBLETON, R. Evaluación psicométrica de los tests informatizados. In: NIELSEN, J. Usability Engineering. California: Morgan Kaufmann, 1993.
NIELSEN, J.; LORANGER, H. Prioritizing Web Usability. California: New Riders, 2006.
OLEA, J. et al. Un test adaptativo informatizado para evaluar el conocimiento de inglés escrito: diseño y comprobaciones psicométricas. Psicothema, v. 16, n. 3, p. 519-525, 2004.
RAU, P.; LIANG, S. F. Internationalization and localization: evaluating and testing a website for Asian users. Ergonomics, v. 46, n. 1, p. 255-270, 2003. http://dx.doi.org/10.1080/00140130303527
RECKASE, M. D. A linear logistic multidimensional model for dichotomous item response data. In: VAN DER LINDEN, W. J.; HAMBLETON, R. K. (Eds.). Handbook of modern item response theory. New York: Springer-Verlag, 1997. p. 271-286.
REISE, S. P.; WIDAMAN, K. F.; PUGH, R. H. Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, v. 114, n. 3, p. 552-566, 1993. http://dx.doi.org/10.1037/0033-2909.114.3.552
ROSSO, M. User-Based Identification of Web Genres. Journal of the American Society for Information Science and Technology, v. 59, n. 5 p. 1-20, 2008.
SANDS, W. A.; WATERS, B. K. Introduction to ASVAB and CAT. In: SANDS, W. A; WATERS, B. K.; MCBRIDE, J. R. Computerized Adaptive Testing: from inquiry to operation. Washington: American Psychological Association, 1997. http://dx.doi.org/10.1037/10244-001
SANTOR, D. A.; RAMSAY, J. O.; ZUROFF, D. C. Nonparametric item analyses of the Beck Depression Inventory: Evaluating gender item bias and response option weights. Psychological Assessment, v. 6, n. 3, p. 255‑70, 1994. http://dx.doi.org/10.1037/1040-3590.6.3.255
SCHENKMAN, B. N.; JÖNSSON, F. U. Aesthetics and preferences of web pages. Behaviour & Information Technology, v. 19, n. 5, p. 367-377, 2000. http://dx.doi.org/10.1080/014492900750000063
SUKAMOLSON, S. Computerized Test/Item Banking and Computerized Adaptive Testing for Teachers and Lecturers. Information Technology and Universities in Asia – ITUA, 2002.
STRAUB, D. W. Validating instruments in MIS research. MIS Quarterly, v. 13, n. 2, p. 147-169, 1989. http://dx.doi.org/10.2307/248922
TAVARES, H. R.; ANDRADE, D. F.; PEREIRA, C. A. Detection of determinant genes and diagnostic via item response theory. Genetics and Molecular Biology, v. 27, n. 4, p. 679-685, 2004. http://dx.doi.org/10.1590/S1415‑47572004000400033
TEJADA, A. J. R. Pasado, presente y futuro de los Tests Adaptativos Informatizados: entrevista con Isaac I. Bejar. Psicothema, v. 13, n. 4, p. 685-690, 2001.
TEZZA, R.; BORNIA, A. C.; ANDRADE, D. F. Measuring web usability using item response theory: Principles, features and opportunities. Interacting with Computers, v. 23, n. 2, p. 167-175, 2011. http://dx.doi.org/10.1016/j.intcom.2011.02.004
TOIT, M. IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Scientific Software International, 2003.
VAN DER LINDEN, W. J.; GLAS, C. A. W. Computerized Adaptive Testing: Theory and Practice. Dordrecht: Kluwer Academic, 2000.
WAINER, H. CATs: Whither and whence. Psicológica: revista de metodología y psicología experimental, v. 21, n. 1, p. 121-133, 2000a.
WAINER, H. Computerized Adaptive Testing: A Primer. New Jersey: Lawrence Erlbaum Associates, 2000b.
Facebook
Google+
Twitter
LinkedIn
Mendeley
StumbleUpon
CiteULike
Reddit
Email