Exploring Confidence Accuracy and Item Difficulty in Changing Multiple-Choice Answers of Scientific Reasoning Test

Sarah Meilani Fadillah; Minsu Ha; Eni Nuraeni; Nurma Yunita  Indriyanti

doi:10.32890/mjli2023.20.2.5

Authors

Sarah Meilani Fadillah Department of Science Education, Kangwon National University, Chuncheon, Korea, Republic of https://orcid.org/0000-0002-3432-1203 (unauthenticated)
Minsu Ha Department of Biology Education, Seoul National University, Seoul, Korea, Republic of
Eni Nuraeni Department of Biology Education, Universitas Pendidikan Indonesia, Bandung, Indonesia https://orcid.org/0000-0003-2852-4730 (unauthenticated)
Nurma Yunita Indriyanti Department of Science Education, Sebelas Maret University, Surakarta, Indonesia https://orcid.org/0000-0002-9113-0235 (unauthenticated)

DOI:

https://doi.org/10.32890/mjli2023.20.2.5

Keywords:

Item difficulty, metacognitive judgement, overconfidence bias, scientific reasoning, confidence accuracy, revision study, undergraduate students

Abstract

Purpose – Researchers discovered that when students were given the opportunity to change their answers, a majority changed their
responses from incorrect to correct, and this change often increased the overall test results. What prompts students to modify their
answers? This study aims to examine the modification of scientific reasoning test, with additional exploration on confidence accuracy
and its relation to item difficulty.

Methodology –A set of pre-test and post-test experiments which included 20 items of scientific reasoning test with confidence
judgement on each item were used. The items of the instruments were assessed for their validity by analysing their psychometric properties using the three-parameter (3PL) Item Response Theory, which was carried out in R studio. The set of items were randomly administered to 205 Indonesian undergraduate students with a background in science education related major. The accuracy of confidence was determined by categorising correct or incorrect answers to scientific reasoning questions based on their level of confidence.

Findings –The results revealed that responses were modified more frequently from incorrect to correct than from correct to incorrect,
resulting in a significant gain in overall scientific reasoning score although these modifications were not shown to be connected to the
item’s difficulty level. Even though confidence level also increased significantly, it was observed that Indonesian students repeatedly
responded with overconfidence even after sitting for the same test after three weeks, which could indicate a lack of metacognitive ability. The findings of this study serve to spur educators to begin actively engaging in metacognitive training in their teaching and learning activities as a result of overconfidence that frequently occurs among Indonesian students in examinations.

Significance –This study provides further substantiation in the field of scientific reasoning and cognitive science; that a trend of confidence accuracy change in scientific reasoning test has been observed. It also contributes to uncovering the true ability of Indonesian students when performing such reasoning tests through their repeated attempts.

References

Ackerman, R., & Thompson, V. A. (2017). Meta-reasoning: Monitoring and control of thinking and reasoning. Trends in Cognitive Sciences, 21(8), 607–617. https://doi.org/10.1016/j. tics.2017.05.004

Archer, N. S., & Pippert, R. (1962). Don’t change the answer! An expose of the perennial myth that first choices are always the correct ones. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 37, 39–41. https://doi.org/10.1080 /00098655.1962.11476207

Ballance, C. T. (1977). Students’ expectations and their answer-changing behavior. Psychological Reports, 41(1), 163–166. https://doi.org/10.2466/pr0.1977.41.1.163

Benjamin, L. T., Jr., Cavell, T. A., & Shallenberger, W. R. (1984). Staying with initial answers on objective tests: Is it a myth? Teaching of Psychology, 11, 133–141. https://doi.org/10.1177/009862838401100303

Bol, L., & Hacker, D. J. (2001). A comparison of the effects of practice tests and traditional review on performance and calibration. The Journal of Experimental Education, 69(2), 133–151. https://doi.org/10.1080/00220970109600653

Brown, G., Irving, E., & Keegan, P. (2008). An introduction to educational assessment, measurement and evaluation. Auckland, NZ: Pearson Education.

Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. Routledge.

Couchman, J. J., Miller, N. E., Zmuda, S. J., Feather, K., & Schwartzmeyer, T. (2016). The instinct fallacy: The metacognition of answering and revising during college exams. Metacognition & Learning, 11, 171–185. https://doi.org/10.1007/s11409-015-9140-8

Coutinho, M. V., Papanastasiou, E., Agni, S., Vasko, J. M., & Couchman, J. J. (2020). Metacognitive monitoring in test-taking situations: A cross-cultural comparison of college students. International Journal of Instruction, 13(1), 407. https://doi.org/10.29333/iji.2020.13127a

Coutinho, M. V., Thomas, J., Lowman, I. F., & Bondaruk, M. V. (2020b). The Dunning-Kruger effect in Emirati college students: Evidence for generalizability across cultures. International Journal of Psychology and Psychological Therapy, 20(1), 29–36.

Coutinho, M. V., Thomas, J., Alsuwaidi, A. S., & Couchman, J. J. (2021). Dunning-Kruger effect: Intuitive errors predict overconfidence on the cognitive reflection test. Frontiers in Psychology, 12, 1040. https://doi.org/10.3389/fpsyg.2021.603225

Craig, K., Hale, D., Grainger, C., & Stewart, M. E. (2020). Evaluating metacognitive self-reports: Systematic reviews of the value of self-report in metacognitive research. Metacognition and Learning, 15(2), 155–213. https://doi.org/10.1007/s11409-020-09222-y

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906

Friedman-Erickson, S. (1994). To change or not to change: The multiple-choice dilemma. Proceedings of the Annual Institute of the American Psychological Society on the Teaching of Psychology, Washington, DC.

Fritzsche, E. S., Kröner, S., Dresel, M., Kopp, B., & Martschinke, S. (2012). Confidence scores as measures of metacognitive monitoring in primary students? (Limited) validity in predicting academic achievement and the mediating role of self-concept. Journal for Educational Research Online, 4(2), 120–142. https://doi.org/10.25656/01:7485

Grant, A. (2021). Think again: The power of knowing what you don’t know. Viking Books.

Ha, M., Sya’bandari, Y., Rusmana, A. N., Aini, R. Q., & Fadillah, S. M. (2021). Comprehensive analysis of the Fort instrument: Using distractor analysis to explore students’ scientific reasoning based on academic level and gender difference. Journal of Baltic Science Education, 20(6), 906. https://doi.org/10.33225/jbse/21.20.906

Harrison, G. M., & Vallin, L. M. (2018). Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition and Learning, 13, 15–38. https://doi.org/10.1007/s11409-017-9176-z

Higham, P. A., & Gerrard, C. (2005). Not all errors are created equal: Metacognition and changing answers on multiple-choice tests. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 59(1), 28. https://doi.org/10.1037/h0087457

Kalinowski, S. T., & Willoughby, S. (2019). Development and validation of a scientific (formal) reasoning test for college students. Journal of Research in Science Teaching, 56(9), 1269–1284. https://doi.org/10.1002/tea.21555

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121

Kruger, J., Wirtz, D., & Miller, D. T. (2005). Counterfactual thinking and the first instinct fallacy. Journal of Personality and Social Psychology, 88(5), 725. https://doi.org/10.1037/0022-3514.88.5.725

Lundeberg, M., & Mohan, L. (2009). Context matters: Gender and cross-cultural differences in confidence. In Handbook of metacognition in education (pp. 221–239). Routledge.

Mata, A., Ferreira, M. B., & Sherman, S. J. (2013). The metacognitive advantage of deliberative thinkers: A dual-process perspective on overconfidence. Journal of Personality and Social Psychology, 105(3), 353. https://doi.org/10.1037/a0033640

May, K., & Jackson, T. S. (2005). IRT item parameters and the reliability and validity of pretest, posttest, and gain scores. International Journal of Testing, 5(1), 63–73.

Merry, J. W., Elenchin, M. K., & Surma, R. N. (2021). Should students change their answers on multiple choice questions? Advances in Physiology Education, 45(1), 182–190. https://doi.org/10.1152/advan.00090.2020

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

Metcalfe, J., & Kornell, N. (2005). A region of proximal learning model of study time allocation. Journal of Memory and Language, 52(4), 463–477. https://doi.org/10.1016/j.jml.2004.12.001

Papanastasiou, E. C. (2015). Psychometric changes on item difficulty due to item review by examinees. Practical Assessment, Research & Evaluation, 20(3), 1–10. https://doi.org/10.7275/ jcyv-k456

Pieschl, S. (2009). Metacognitive calibration—an extended conceptualization and potential applications. Metacognition and Learning, 4(1), 3–31. https://doi.org/10.1007/s11409-008-9030-4

Rachmatullah, A., & Ha, M. (2019). Examining high-school students’ overconfidence bias in biology exam: A focus on the effects of country and gender. International Journal of Science Education, 41(5), 652–673. https://doi.org/10.1080/09500693.2019.1578002

Rusmana, A. N., Roshayanti, F., & Ha, M. (2020). Debiasing overconfidence among Indonesian undergraduate students in the biology classroom: An intervention study of the KAAR model. Asia-Pacific Science Education, 6(1), 228–254. https://doi.org/10.1163/23641177-BJA00001

Rutherford, T. (2017). The measurement of calibration in real contexts. Learning and Instruction, 47, 33–42. https://doi.org/10.1016/j. learninstruc.2016.10.006

Schraw, G., Kuch, F., & Gutierrez, A. P. (2013). Measure for measure: Calibrating ten commonly used calibration scores. Learning and Instruction, 24, 48–57. https://doi.org/10.1016/j. learninstruc.2012.08.007

Stone, C., Mattingley, J. B., & Rangelov, D. (2022). On second thoughts: Changes of mind in decision-making. Trends in Cognitive Sciences. https://doi.org/10.1016/j.tics.2022.02.004

Stylianou-Georgiou, A., & Papanastasiou, E. C. (2017). Answer changing in testing situations: The role of metacognition in deciding which answers to review. Educational Research and Evaluation, 23(3–4), 102–118. https://doi.org/10.1080/138036 11.2017.1390479

Wagner, D., Cook, G., & Friedman, S. (1998). Staying with their first impulse? The relationship between impulsivity/reflectivity, field dependence/field independence and answer changes on a multiple-choice exam in a fifth-grade sample. Journal of Research and Development in Education, 31, 166–175.

Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.