Students’ mathematical justification abilities in analyzing ChatGPT’s answers

Joko Sutrisno AB; Elvandri Yogi Pratama; Andika Putra R; Syaiful Hamzah Nasution; Latifah Mustofa Lestyanto

doi:10.22460/infinity.v14i2.p445-460

Authors

Joko Sutrisno AB Department of Mathematics Education, STKIP-PGRI Bandar Lampung https://orcid.org/0009-0008-2335-764X
Elvandri Yogi Pratama Department of Mathematics Education, STKIP-PGRI Bandar Lampung https://orcid.org/0000-0002-1624-7177
Andika Putra R Department of Mathematics Education, Universitas Tamansiswa https://orcid.org/0009-0007-2707-6864
Syaiful Hamzah Nasution Department of Mathematics Education, Universitas Negeri Malang https://orcid.org/0000-0003-2089-0285
Latifah Mustofa Lestyanto School of Mathematical and Statistical Sciences, University of Galway

DOI:

https://doi.org/10.22460/infinity.v14i2.p445-460

Keywords:

Artificial intelligence, ChatGPT, Mathematical justification ability

Abstract

This study investigates students' mathematical justification abilities in analyzing answers obtained from one of the Artificial Intelligence (AI) based services, namely ChatGPT. The subjects of this descriptive qualitative study are 13 fourth-semester students of STKIP-PGRI Bandar Lampung. They were asked to utilize ChatGPT to solve two linear equation system questions adopted from Indonesian university entrance tests in 2019. The students were then requested to analyze and justify the responses generated by ChatGPT. The justifications from students were then analyzed to see their abilities and determine whether the answers from ChatGPT were correct. As many as 61.54% of the respondents could understand the instructions of the two questions and provide reviews for the answers generated by ChatGPT. 38.46% of the respondents did not understand the questions' instructions. The results showed that students may have lacked the necessary ability to analyze the AI-generated answers in depth and explain them in detail. This may be due to weak mastery of the basic concepts of the material and a lack of accuracy. The findings call for paying more attention to students' mathematical justification abilities in the current AI-assisted era.

References

Baidoo-anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52-62. https://doi.org/10.61969/jai.1337500

Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLoS One, 18(8), e0290691. https://doi.org/10.1371/journal.pone.0290691

Farhi, F., Jeljeli, R., Aburezeq, I., Dweikat, F. F., Al-shami, S. A., & Slamene, R. (2023). Analyzing the students' views, concerns, and perceived ethics about chat GPT usage. Computers and Education: Artificial Intelligence, 5, 100180. https://doi.org/10.1016/j.caeai.2023.100180

Fatmanissa, N., Siswono, E., Yuli, T., Lukito, A., & Ismail, I. (2024). Utilización de una tarea de toma de decisiones: justificación matemática de los estudiantes en la resolución colaborativa de problemas [Utilizing decision-making task: Students’ mathematical justification in collaborative problem solving]. Avances de investigación en Educación Matemática, 26, 85-103. https://doi.org/10.35763/aiem26.5341

Fijačko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023). Can ChatGPT pass the life support exams without entering the American heart association course? Resuscitation, 185, 109732. https://doi.org/10.1016/j.resuscitation.2023.109732

García-López, I. M., González González, C. S., Ramírez-Montoya, M.-S., & Molina-Espinosa, J.-M. (2025). Challenges of implementing ChatGPT on education: Systematic literature review. International Journal of Educational Research Open, 8, 100401. https://doi.org/10.1016/j.ijedro.2024.100401

Govender, R. (2023). The impact of artificial intelligence and the future of ChatGPT for mathematics teaching and learning in schools and higher education. Pythagoras, 44(1), a787. https://doi.org/10.4102/pythagoras.v44i1.787

Hanna, G., & Jahnke, H. N. (1996). Proof and proving. In A. J. Bishop, K. Clements, C. Keitel, J. Kilpatrick, & C. Laborde (Eds.), International handbook of mathematics education (pp. 877-908). Springer Netherlands. https://doi.org/10.1007/978-94-009-1465-0_24

Hershkowitz, R. (2020). Shape and space: Geometry teaching and learning. In S. Lerman (Ed.), Encyclopedia of mathematics education (pp. 774-779). Springer International Publishing. https://doi.org/10.1007/978-3-030-15789-0_138

Holmes, W., Bialik, M., & Fadel, C. (2023). Artificial intelligence in education. In Data ethics : building trust : how digital technologies can serve humanity (pp. 621-653). Globethics Publications. https://doi.org/10.58863/20.500.12424/4276068

Iqbal, M., Khan, N. U., & Imran, M. (2024). The role of artificial intelligence (AI) in transforming educational practices: Opportunities, challenges, and implications. Qlantic Journal of Social Sciences, 5(2), 348-359. https://doi.org/10.55737/qjss.349319430

Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198

Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410

OECD. (2018). Education at a Glance 2018: OECD Indicators. OECD Publishing. https://doi.org/10.1787/eag-2018-en

OECD. (2022). Trends Shaping Education 2022. OECD Publishing. https://doi.org/10.1787/6ae8771a-en

Ojeda, A. D., Solano-Barliza, A. D., Alvarez, D. O., & Cárcamo, E. B. (2023). Análisis del impacto de la inteligencia artificial ChatGPT en los procesos de enseñanza y aprendizaje en la educación universitaria [Analysis of the impact of artificial intelligence ChatGPT on the teaching and learning processes in university education]. Formación universitaria, 16(6), 61-70. https://doi.org/10.4067/S0718-50062023000600061

Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/10.3390/app13095783

Santos, L., Mata-Pereira, J., da Ponte, J. P., & Oliveira, H. (2022). Teachers’ understanding of generalizing and justifying in a professional development course. Eurasia Journal of Mathematics, Science and Technology Education, 18(1), em2067. https://doi.org/10.29333/ejmste/11488

Sarumaha, Y. A., & Rizkianto, I. (2022). Promoting mathematical justification through realistic mathematics education classroom. Jurnal Pendidikan Matematika (JUPITEK), 5(2), 83-94. https://doi.org/10.30598/jupitekvol5iss2pp83-94

Sukirwan, S., Muhtadi, D., Saleh, H., & Warsito, W. (2020). Profile of students' justifications of mathematical argumentation. Infinity Journal, 9(2), 197-212. https://doi.org/10.22460/infinity.v9i2.p197-212

Vasconcelos, M. A. R., & Santos, R. P. d. (2023). Enhancing STEM learning with ChatGPT and Bing Chat as objects to think with: A case study. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2296. https://doi.org/10.29333/ejmste/13313

Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2286. https://doi.org/10.29333/ejmste/13272

Wei, X., Chu, X., Geng, J., Wang, Y., Wang, P., Wang, H., Wang, C., & Lei, L. (2024). Societal impacts of chatbot and mitigation strategies for negative impacts: A large-scale qualitative survey of ChatGPT users. Technology in Society, 77, 102566. https://doi.org/10.1016/j.techsoc.2024.102566

Welding, L. (2023). Half of college students say using AI on schoolwork is cheating or plagiarism. BestColleges. Retrieved from https://www.bestcolleges.com/research/college-students-ai-tools-survey

Xu, X., Su, Y., Zhang, Y., Wu, Y., & Xu, X. (2024). Understanding learners' perceptions of ChatGPT: A thematic analysis of peer interviews among undergraduates and postgraduates in China. Heliyon, 10(4), e26239. https://doi.org/10.1016/j.heliyon.2024.e26239

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0

Zhai, X. (2022). ChatGPT: Artificial Intelligence for Education. Retrieved from https://doi.org/10.13140/RG.2.2.35971.37920