COMPUTERIZED ADAPTIVE TEST (CAT) OPTIMIZATION USING THE EFFICIENCY BALANCED INFORMATION (EBI) METHOD

Aji Joko Budi Pramono* -  Institut Agama Islam Negeri Ternate, Indonesia

This study aims to optimize Efficiency Balanced Information (EBI) for selecting items in the computer-based adaptive exam model (CAT), usually used in education. This optimization is used to reduce problems that often occur in CAT based on item response theory (TRB), namely the emergence of ability bias, large Mean Absolute Error (MAE), too many test lengths, and uncontrolled Standard Error Estimation (SEE). The success of the CAT-based exam is strongly influenced by the item selection method; therefore, an appropriate item selection strategy is needed. This study proposes selecting items using the EBI method to solve this problem. The simulation results show that the grain bias is slightly around zero, and CMAE is consistent in the allowable theta area. The average number of items presented is less than 20 when -2<θ<2. The average test length increases to 25 seconds | θ | > 2. The items presented show the different power parameters of questions (a) at the beginning of the test more than at the end of the test; therefore, this method can be the right solution for optimizing accuracy in selecting items

  1. Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4). https://doi.org/10.1007/BF02293801
  2. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP Estimation of Ability in a Microcomputer Environment. Applied Psychological Measurement, 6(4). https://doi.org/10.1177/014662168200600405
  3. Bunderson, C. V., Inouye, D. K., & Olsen, J. B. (1988). THE FOUR GENERATIONS OF COMPUTERIZED EDUCATIONAL MEASUREMENT. ETS Research Report Series, 1988(1), i–148. https://doi.org/10.1002/j.2330-8516.1988.tb00291.x
  4. Chrysafiadi, K., Troussas, C., & Virvou, M. (2020). Combination of fuzzy and cognitive theories for adaptive e-assessment. Expert Systems with Applications, 161. https://doi.org/10.1016/j.eswa.2020.113614
  5. Deng, H., Ansley, T., & Chang, H. H. (2010). Stratified and maximum information item selection procedures in computer adaptive testing. Journal of Educational Measurement, 47(2). https://doi.org/10.1111/j.1745-3984.2010.00109.x
  6. Diana, H., & Maula, R. (2022). Manajemen Sumber Daya Manusia Dalam Meningkatkan Kinerja Organisasi Panji Pelopor di Pondok Pesantren. JUMPA: Jurnal Manajemen Pendidikan, 3(1).
  7. Dunkel, P. A. (1998). Considerations in developing or using second/foreign language proficiency computer-adaptive tests. Language Learning and Technology, 2(2).
  8. Falk, C. F., & Feuerstahler, L. M. (2022). On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests. Educational and Psychological Measurement, 82(1). https://doi.org/10.1177/00131644211014261
  9. Gawliczek, P., Krykun, V., Tarasenko, N., Tyshchenko, M., & Shapran, O. (2021). COMPUTER ADAPTIVE LANGUAGE TESTING ACCORDING TO NATO STANAG 6001 REQUIREMENTS. Advanced Education, 8(18). https://doi.org/10.20535/2410-8286.225018
  10. Gibbons, C., Bachulis, A., & Alien, G. (1999). A comparison of a computer and pencil and paper assignment. Computers in Nursing, 17(6).
  11. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Sage Publications.
  12. Han, K. C. T. (2018a). Components of the item selection algorithm in computerized adaptive testing. In Journal of educational evaluation for health professions (Vol. 15). https://doi.org/10.3352/jeehp.2018.15.7
  13. Han, K. C. T. (2018b). Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece. Journal of Educational Evaluation for Health Professions, 15. https://doi.org/10.3352/jeehp.2018.15.20
  14. Hussain, J., Chawla, G., Rafiqzad, H., Huang, S., Bartlett, S. J., Li, M., Howell, D., Peipert, J. D., Novak, M., & Mucsi, I. (2022). Validation of the PROMIS sleep disturbance item bank computer adaptive test (CAT) in patients on renal replacement therapy. Sleep Medicine, 90. https://doi.org/10.1016/j.sleep.2022.01.001
  15. International Test Comission. (2005). Guidelines for Translating and Adapting Tests. International Test Commission (ITC).
  16. Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4). https://doi.org/10.1207/s15324818ame0204_6
  17. LAN, X., YU, H., & LI, J. (2017). Optimization of Moodle for Adaptive Testing. DEStech Transactions on Computer Science and Engineering, cmsam. https://doi.org/10.12783/dtcse/cmsam2017/16420
  18. Li, G., Cai, Y., Gao, X., Wang, D., & Tu, D. (2021). Automated Test Assembly for Multistage Testing With Cognitive Diagnosis. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.509844
  19. Lord, F. M. (2012). Applications of item response theory to practical testing problems. In Applications of Item Response Theory To Practical Testing Problems. https://doi.org/10.4324/9780203056615
  20. Marastuti, A., Anggoro, W. J., Marvianto, R. D., & Al Afghani, A. A. (2020). Perbandingan Properti Psikometri antara Tes PAPs Berbentuk Computer-Based dan Paper and Pencil Test. Gadjah Mada Journal of Psychology (GamaJoP), 6(1). https://doi.org/10.22146/gamajop.51852
  21. Maravić Čisar, S., Čisar, P., & Pinter, R. (2016). Evaluation of knowledge in Object Oriented Programming course with computer adaptive tests. Computers and Education, 92–93. https://doi.org/10.1016/j.compedu.2015.10.016
  22. Naeem, F. (2019). Cultural adaptations of CBT: A summary and discussion of the Special Issue on Cultural Adaptation of CBT. Cognitive Behaviour Therapist. https://doi.org/10.1017/S1754470X19000278
  23. Oud, M., de Winter, L., Vermeulen-Smit, E., Bodden, D., Nauta, M., Stone, L., van den Heuvel, M., Taher, R. Al, de Graaf, I., Kendall, T., Engels, R., & Stikkelbroek, Y. (2019). Effectiveness of CBT for children and adolescents with depression: A systematic review and meta-regression analysis. In European Psychiatry (Vol. 57). https://doi.org/10.1016/j.eurpsy.2018.12.008
  24. Owen, R. J. (1975). A bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70(350). https://doi.org/10.1080/01621459.1975.10479871
  25. Popova, Y. B. (2019). Learning Management System CATS. Science & Technique, 18(4). https://doi.org/10.21122/2227-1031-2019-18-4-339-349
  26. Postigo, Á., Cuesta, M., Pedrosa, I., Muñiz, J., & García-Cueto, E. (2020). Development of a computerized adaptive test to assess entrepreneurial personality. Psicologia: Reflexao e Critica, 33(1). https://doi.org/10.1186/s41155-020-00144-x
  27. Quiñones, A. C., & Humphrey, G. E. (2007). Addressing the pharmacist shortage through a cooperative internship program for foreign pharmacy graduates. Journal of the American Pharmacists Association, 47(2). https://doi.org/10.1331/QJ0N-4278-738U-0061
  28. Rezaie, M., & Golshan, M. (2015). Computer Adaptive Test (CAT): Advantages and Limitations. International Journal of Educational Investigations Available Online @ Www.Ijeionline.Com, 2(5).
  29. Solberg, B. L. (2015). Critical Thinking as a Predictor of Certification Exam Performance in Medical Laboratory Science. American Society for Clinical Laboratory Science, 28(2). https://doi.org/10.29074/ascls.28.2.76
  30. Sudaryono. (2011). Implementasi Teori Responsi Butir (Item Response Theory) Pada Penilaian Hasil Belajar Akhir di Sekolah. Jurnal Pendidikan Dan Kebudayaan, 17(6). https://doi.org/10.24832/jpnk.v17i6.62
  31. Tisocco, F., & Liporace, M. F. (2021). The tuckman procrastination scale: Psychometric features among Buenos Aires undergraduates. Psychological Thought, 14(2). https://doi.org/10.37708/psyct.v14i2.603