Psychometric Evaluation and Item Response Theory Validation of a Higher-Order Thinking Skills Test on Buffer Solution Concepts in Chemistry
DOI:
https://doi.org/10.32890/mjli2026.23.2.7Keywords:
Higher-order thinking skills, buffer solutions, chemistry education, item response theory, generalized partial credit model, test validationAbstract
Purpose – This study aims to evaluate and validate a higher-order thinking skills (HOTS) test instrument for buffer solution concepts using the generalized partial credit model (GPCM). While HOTS is critical in chemistry, there is a lack of specialized instruments validated against modern psychometric theories to accurately assess students’ analytical and evaluative abilities with regard to buffer solutions.
Methodology – A quantitative research approach was employed. The instrument used was a test instrument with an 8-item polytomous instrument. It was administered to 288 high school students in Bengkulu City, Indonesia. Data were analyzed using an item response theory (IRT) - GPCM to estimate item parameters (discrimination and difficulty) and test information function.
Findings – The results confirmed that 7 out of 8 items exhibited good fit to the GPCM. These items cover a wide range of difficulty levels, effectively distinguishing between students with varying abilities in HOTS. The test provided maximum information at slightly above-average abilities (θ ≈ 0.49), and remained across the low to high ability range, making it a precise tool for mapping a wide variation in student abilities in HOTS, although less than optimal for truly extreme abilities.
Novelty – This study bridges modern psychometrics and chemistry pedagogy by utilizing the GPCM to map student cognitive load during multi-step buffer solution reasoning. Unlike previous assessments limited to a standard ability range, the instrument used crossed the cognitive threshold and offered teachers a high-precision assessment tool.
Significance – The study contributes to the field of chemistry education by providing a psychometrically sound instrument for diagnosing HOTS. Unlike traditional classical test theory, the application of the GPCM allows teachers to identify specific conceptual gaps in student reasoning regarding buffer solutions, enabling more targeted instructional interventions.
References
Algouzi, S., Alzubi, A. A. F., & Nazim, M. (2023). Enhancing EFL students’ critical thinking skills using a technology-mediated self-study approach: EFL graduates and labor market in perspective. PLoS ONE, 18(10), e0293273. https://doi.org/10.1371/journal.pone.0293273
Alkhatib, O. J. (2019). A framework for implementing higher-order thinking skills (problem-solving, critical thinking, creative thinking, and decision-making) in engineering humanities. 2019 Advances in Science and Engineering Technology International Conferences (ASET), 8714262. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICASET.2019.8714232
Anderson, L. W., & Krathwohl, D. R., (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814
Baker, F. B. & Kim, S.-H. (2017). The basics of item response theory: Statistics for social and behavioral sciences. Springer International Publishing.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
Bolt, D. M., & Adams, D. J. (2016). Exploring rubric-related multidimensionality in polytomously scored test items. Applied Psychological Measurement, 41(3), 163–177. https://doi.org/10.1177/0146621616677715
Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06
Chamidah, D., Suwono, H., Susilo, J., Sulisetijono, S., Munzil, M., & Zubaedah, S. (2024). Navigating the global landscape on project-based learning and 21st-century skills research (2020-2023): A bibliometric analysis. Malaysian Journal of Learning and Instruction, 21(1), 217–247. https://doi.org/10.32890/mjli2024.21.2.7
Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.3102/10769986022003265
Cheng, T. C. E., & Choy, P. W. C. (2007). Measuring success factors of quality management in the shipping industry. Maritime Economics and Logistics, 9(3), 234–253. https://doi.org/10.1057/palgrave.mel.9100179
Chi, S., Wang, Z., & Liu, X. (2023). Assessment of context-based chemistry problem-solving skills: Test design and results from ninth-grade students. Research in Science Education, 53(2), 295–318. https://doi.org/10.1007/s11165-022-10056-8
Clarke, D., & Roche, A. (2018). Using contextualized tasks to engage students in meaningful and worthwhile mathematics learning. Journal of Mathematical Behavior, 51, 95–108. https://doi.org/10.1016/j.jmathb.2017.11.006
Dai, S., Vo, T. T., Kehinde, O. J., He, H., Xue, Y., Demir, C., & Wang, X. (2021). Performance of polytomous IRT models with rating scale data: An investigation over sample size, instrument length, and missing data. Frontiers in Education, 6, 1–18. https://doi.org/10.3389/feduc.2021.721963
de Ayala, R. J. (2022). The theory and practice of item response theory (2nd eds). The Guilford Press.
Eliaumra, E., Samaela, D. P., & Muhdin, N. K. (2022). Developing diagnostic test assessment to measure creative thinking skills of Biology preservice teacher students. REID (Research and Evaluation in Education), 8(2), 152–168. https://doi.org/10.21831/reid.v8i2.50885
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates Inc. https://doi.org/10.4324/9781410605269
Finch, H., & French, B. F. (2019). A comparison of estimation techniques for IRT models with small samples. Applied Measurement in Education, 32(2), 77–96. https://doi.org/10.1080/08957347.2019.1577243
FitzPatrick, B., & Schulz, H. (2015). Do curriculum outcomes and assessment activities in science encourage higher order thinking? Canadian Journal of Science, Mathematics and Technology Education, 15(2), 136–154. https://doi.org/10.1080/14926156.2015.1014074
Fox, J., Weisberg, S., & Price, B. (2023). car: Companion to Applied Regression. https://cran.r-project.org/web/packages/car/index.html
García-Pérez, M. A. (2014). Multiple-choice tests: Polytomous IRT models misestimate item information. The Spanish Journal of Psychology, 17, 88. https://doi.org/10.1017/sjp.2014.95
Georgiev, N. (2008). Item analysis of C, D and E series from Raven’s standard progressive matrices with item response theory two-parameter logistic model. Europe’s Journal of Psychology, 4(3), e431. https://doi.org/10.5964/ejop.v4i3.431
Hai, T. N., & Thi, H. N. (2021). Training for critical thinking for high school students to meet the requirements of education innovation. Journal of Science Educational Science, 66(1), 46–56. https://doi.org/10.18173/2354-1075.2021-0005
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item respons (pp. 1–174). Sage Publication.
Harsana, F. N., Retnawati, H., Dewanti, S. R., & Lumenyela, R. A. (2024). Comparison of item characteristic analysis models of reading literacy test with polytomous item response theory. REID (Research and Evaluation in Education), 10(2), 214–226. https://doi.org/10.21831/reid.v10i2.77852
Hartono, W., Hadi, S., Rosnawati, R., & Retnawati, H. (2023). Exploration of student’ cognitive mathematics ability diagnostic instruments: Validity, reliability, and item characteristics. Pegem Journal of Education and Instruction, 13(3), 386–394. https://doi.org/10.47750/pegegog.13.03.39
Hendrickson, A., & Mislevy, R. J. (2005). Item response theory: Cognitive models. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of statistics in behavioral science (pp. 978–982). Wiley. https://doi.org/10.1002/0470013192.bsa108
Hillsley, K. (2025). Question format is the best predictor of item discrimination: A multivariable analysis. Journal of Microbiology & Biology Education, 26(3). https://doi.org/10.1128/jmbe.00205-25
Ismail, S. N., Muhammad, S., Omar, M. N., & Shanmugam, K. S. (2022). The practice of critical thinking skills in teaching mathematics: Teachers’ perception and readiness. Malaysian Journal of Learning and Instruction, 19(1), 1–30. https://doi.org/10.32890/mjli2022.19.1.1
Jo, D., Im, S., Suh, D. E., Spencer, S. D., & Masuda, A. (2023). The personalized psychological flexibility index (PPFI): An item response theory analysis with racially diverse college students. Journal of Psychopathology and Behavioral Assessment, 45(3), 829–843. https://doi.org/10.1007/s10862-023-10076-2
Kang, T., Cohen, A. S., & Sung, H. J. (2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33(7), 499–518. https://doi.org/10.1177/0146621608327800
Kassambara, A., & Mundt, F. (2020). factoextra: Extract and visualize the results of multivariate data analyses (1.0.7). CRAN. https://doi.org/10.32614/CRAN.package.factoextra
Kim, J., & Wilson, M. (2019). Polytomous item explanatory item response theory models. Educational and Psychological Measurement, 80(4), 726–755. https://doi.org/10.1177/0013164419892667
Lê, S., Josse, J., & Husson, F. (2008). FactoMineR: An R package for multivariate analysis. Journal of Statistical Software, 25(1), 1–18. https://doi.org/10.18637/jss.v025.i01
Lubbe, D. (2023). Advantages of using unweighted approximation error measures for model fit assessment. Psychometrika, 88(2), 413–433. https://doi.org/10.1007/s11336-023-09909-6
Marlina, Hadi, S., & Rahim, A. (2025). The application of item response theory for development tests of high order thinking skills on chemical learning: Partial credit models study. International Journal of Educational Reform, 0(0). https://doi.org/10.1177/10567879251323884
Maryani, I., Prasetyo, Z. K., Wilujeng, I., & Purwanti, S. (2022). Higher-order thinking test of science for college students using multidimensional item response theory analysis. Pegem Egitim ve Ogretim Dergisi, 12(1), 292–300. https://doi.org/10.47750/pegegog.12.01.30
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272
Maydeu-Olivares, A., & Joe, H. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075
Mellenbergh, G. J. (2014). Item bias detection: Modern approaches. In N. Balakrishnan, T. Colton, B. Everitt, W. Piegorsch, F. Ruggeri & J. L. Teugels (Eds.), Wiley StatsRef: Statistics Reference Online (pp. 1–6). John Wiley & Sons. https://doi.org/10.1002/9781118445112.stat06390
Muntari, M., Hadisaputra, S., Haris, M., Loka, I. N., Hayati, H., & Yanti, E. (2021). Optimization of achievement in chemistry learning outcomes by implementing discovery learning-based strategy. Proceeding of the 2nd Annual Conference on Education and Social Science (ACCESS 2020), 274-278. https://doi.org/10.2991/assehr.k.210525.089
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–176. https://doi.org/10.1177/014662169201600206
Muraki, E. (1997). A generalized partial credit model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 153–164). Springer. https://doi.org/10.1007/978-1-4757-2691-6
Nunes, S., Oliveira, T., & Oliveira, A. (2017). Item response theory – a first approach. AIP Conference Proceeding, 1863(1). https://doi.org/10.1063/1.4992683
Nye, C. D., Joo, S. H., Zhang, B., & Stark, S. (2020). Advancing and evaluating IRT model data fit indices in organizational research. Organizational Research Methods, 23(3), 457–486. https://doi.org/10.1177/1094428119833158
Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003
Pada, A. U. T., Mustakim, S. S., & Subali, B. (2018). Construct validity of creative thinking skills instrument for biology student teachers in the subject of human physiology. Jurnal Penelitian dan Evaluasi Pendidikan, 22(2), 119–129. https://doi.org/10.21831/pep.v22i2.22369
Pardede, T., Santoso, A., Diki, D., Retnawati, H., Rafi, I., Apino, E., & Rosyada, M. N. (2023). Gaining a deeper understanding of the meaning of the carelessness parameter in the 4PL IRT model and strategies for estimating it. REID (Research and Evaluation in Education), 9(1), 86–117. https://doi.org/10.21831/reid.v9i1.63230
Penfield, R. D. (2010). Explaining crossing DIF in polytomous items using differential step functioning effects. Applied Psychological Measurement, 34(8), 563–579. https://doi.org/10.1177/0146621610377083
Penfield, R. D. (2014). An NCME instructional module on polytomous item response theory models. Educational Measurement Issues and Practice, 33(1), 36–48. https://doi.org/10.1111/emip.12023
Phakiti, A. (2018). Assessing higher‐order thinking skills in language learning. In J. I. Liontas (Ed.), The TESOL encyclopedia of English language teaching. John Wiley & Son. https://doi.org/10.1002/9781118784235.eelt0380
Pratama, G. S., & Retnawati, H. (2018). Urgency of higher order thinking skills (HOTS) content analysis in mathematics textbook. Journal of Physics: Conference Series, 1097(1). https://doi.org/10.1088/1742-6596/1097/1/012147
Pratomo, H., Fitriyana, N., Wiyarsi, A., & Marfuatun. (2025). Mapping chemistry learning difficulties of secondary school students: A cross-grade study. Journal of Education and Learning, 19(2), 909–920. https://doi.org/10.11591/edulearn.v19i2.21826
Preston, K. S. J., & Reise, S. P. (2013). Estimating the nominal response model under nonnormal conditions. Educational and Psychological Measurement, 74(3), 377–399. https://doi.org/10.1177/0013164413507063
Qiu, X.-L., & Torre, J. d. l. (2023). A dual process item response theory model for polytomous multidimensional forced‐choice items. British Journal of Mathematical and Statistical Psychology, 76(3), 491–512. https://doi.org/10.1111/bmsp.12303
Rahman, T. (2019). Peningkatan kemampuan berfikir kritis matematis siswa melalui model pembelajaran berbasis masalah. Symmetry: Pasundan Journal of Research in Mathematics Learning and Education, 4(2), 72–80. https://doi.org/10.23969/symmetry.v4i2.2085
Rampean, B. A. O., & Rohaeti, E. (2025). The development of an integrated instrument to measure higher order thinking skills and scientific attitudes. Journal of Turkish Science Education, 22(1), 48–62. https://doi.org/10.36681/tused.2025.004
Reise, S. P. (2014). Item response theory. In R.L. Cautin & S.O. Lilienfeld (Eds.), The encyclopedia of clinical psychology (pp.1–10). John Wiley & Son. https://doi.org/10.1002/9781118625392.wbecp357
Reise, S. P. (2016). The emergence of item response theory models and the patient reported outcomes measurement information systems. Austrian Journal of Statistics, 38(4), 211-220. https://doi.org/10.17713/ajs.v38i4.274
Retnawati, H. (2014). Teori respons butir dan penerapannya: Untuk peneliti, praktisi pengukuran dan pengajian mahasiswa dan pascasarjana. Nuha Medika.
Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research. https://cran.r-project.org/package=psych
Rizopoulos, D. (2006). ltm: An R package for latent variable modeling and item response theory analyses. Journal of Statistical Software, 17(5), 1–25. https://doi.org/10.18637/jss.v017.i05
Robitzsch, A., Kiefer, T., & Wu, M. (2023). TAM: Test analysis modules. https://cran.r-project.org/package=TAM
Rohim, A. A., Suharti, S., & Herunata. (2025). Enhancing higher order thinking skills (HOTS) through a scientific approach oriented in teaching salt hydrolysis. Journal of Physics: Conference Series, 3148(1). https://doi.org/10.1088/1742-6596/3148/1/012027
Saepuzaman, D., Istiyono, E., & Haryanto. (2022). Characteristics of fundamental physics higher-order thinking skills test using item response theory analysis. Pegem Egitim ve Ogretim Dergisi, 12(4), 269–279. https://doi.org/10.47750/pegegog.12.04.28
Salgado, F. J. A. (2017). The role of context and context familiarity on mathematics problems. Revista Latinoamericana de Investigacion En Matematica Educativa, 20(3), 265–292. https://doi.org/10.12802/relime.17.2031
Samejima, F. (1997). Graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 85–100). Springer.
Schauberger, P., & Walker, A. (2025). openxlsx: Read, write and edit xlsx files (4.2.8.1). CRAN. https://doi.org/10.32614/CRAN.package.openxlsx
Shanmugam, S. K. S., & Rajoo, M. (2020). Examining the quality of English test items using psychometric and linguistic characteristics among grade six pupils. Malaysian Journal of Learning and Instruction, 17(2), 63–101. https://doi.org/10.32890/mjli2020.17.2.3
Stanke, L., & Bulut, O. (2019). Explanatory item response models for polytomous item responses. International Journal of Assessment Tools in Education, 6(2), 259–278. https://doi.org/10.21449/ijate.515085
Stojanov, A. (2018). Application of item response theory to psychological test construction. Psychology Science and Practice, 4, 67–82. https://doi.org/10.37509/psyscip184067s
Su, K. (2021). A new assessment of HOCS-oriented learning for students’ higher order thinking abilities by Marzano’s Taxonomy. Journal of Baltic Science Education, 20(2), 305–315. https://doi.org/10.33225/jbse/21.20.305
Sumintono, B. (2015). Aplikasi pemodelan Rasch: Pada assessment pendidikan. Penerbit Trim Komunikata.
Sun, J., Xin, T., Zhang, S., & Torre, J. d. l. (2013). A polytomous extension of the generalized distance discriminating method. Applied Psychological Measurement, 37(7), 503–521. https://doi.org/10.1177/0146621613487254
Suswanti, N., Solikhin, F., & Ginting, S. M. (2024). Development of cognitive assessment instruments on buffer solution material based on HOTS (higher order thinking skills). Journal of Chemistry Education Research, 8(2), 97–103. https://doi.org/10.26740/jcer.v8n2.p97-103
Swaminathan, H., Hambleton, R. K., & Rogers, H. J. (2006). Assessing the fit of item response theory models. In C.R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 683–718). Elsevier. https://doi.org/10.1016/S0169-7161(06)26021-8
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295–312. https://doi.org/10.1016/0959-4752(94)90003-5
Tan, T. K. (2024). Evaluating assessment via item response theory utilizing information function with R. The Quantitative Methods for Psychology, 20(1), 33–49. https://doi.org/10.20982/tqmp.20.1.p033
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Triastuti, T., & Laksono, E. W. (2021). Validation of critical thinking test to senior high school students. Journal of Physics Conference Series, 1806(1), 12201. https://doi.org/10.1088/1742-6596/1806/1/012201
Vidergor, H. E. (2018). Effectiveness of the multidimensional curriculum model in developing higher-order thinking skills in elementary and secondary students. Curriculum Journal, 29(1), 95–115. https://doi.org/10.1080/09585176.2017.1318771
Wallmark, J., Ramsay, J. O., Li, J., & Wiberg, M. (2024). Analyzing polytomous test data: A comparison between an information-based IRT model and the generalized partial credit model. Journal of Educational and Behavioral Statistics, 49(5), 753–779. https://doi.org/10.3102/10769986231207879
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org
Wickham, H., François, R., Henry, L., Müller, K., & Vaughan, D. (2026). dplyr: A grammar of data manipulation (1.2.0). CRAN. https://doi.org/10.32614/CRAN.package.dplyr
Widiantie, R., Handayani, H., Setiawati, I., & Roqiqulqolby, A. (2022). Development of higher order thinking skills assessment based on scientific article reviews. Proceeding of the 2nd Universitas Kuningan International Conference on System, Engineering, and Technology (UNISET 2021). EAI. https://doi.org/10.4108/eai.2-12-2021.2320320
Wildan, W., Supriadi, S., Laksmiwati, D., & Analita, R. N. (2021). Environmental chemistry course assisted problem-based learning in developing students’ higher-order thinking skills and characters. Acta Chimica Asiana, 4(2), 141–146. https://doi.org/10.29303/aca.v4i2.54
Willse, J. T. (2026). CTT: Classical test theory functions (2.3.4). CRAN. https://cran.r-project.org/package=CTT
Xia, Y., & Yang, Y. (2019). RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods. Behavior Research Methods, 51(1), 409–428. https://doi.org/10.3758/s13428-018-1055-2
Yosepha, A., Ali, M., Wahyudin, D., & Rusman. (2023). The role of multi-dimensional curriculum design in improving higher-order thinking skills. International Journal of Learning, Teaching and Educational Research, 22(7), 219–239. https://doi.org/10.26803/ijlter.22.7.12
Downloads
Published
Issue
Section
How to Cite
Research impact
Harvested 2026-11-05Counts differ between services because each indexes a different body of literature. None of them is the whole picture.
