User Evaluation of Conversational AI Systems: The Roles of Digital Literacy, Trust, and Perceived Privacy Risk

Authors

  • Guan Xian School of Arts, Universiti Sains Malaysia, Malaysia
  • Norfarizah Mohd Bakhi School of Arts, Universiti Sains Malaysia, Malaysia

DOI:

https://doi.org/10.32890/jict2026.25.4.3

Keywords:

Digital literacy, trust, perceived privacy risk, conversational AI, user evaluation

Abstract

Conversational AI is spreading rapidly, and understanding how users evaluate these systems has become an important issue in information systems research. Drawing on an information systems perspective, this study develops a structural model integrating digital literacy, trust, and perceived privacy risk to examine how digital literacy influences users’ evaluations of conversational AI systems through key psychological mechanisms. Using survey data collected from 580 Chinese Generation Z users, the study employs PLS-SEM to test the proposed model. The results indicate that digital literacy significantly enhances trust, which in turn positively influences system evaluation. Digital literacy is also positively associated with perceived privacy risk, suggesting that more digitally literate users are more sensitive to data practices and system uncertainty. However, perceived privacy risk does not significantly affect system evaluation, indicating that users tend to balance potential risks against perceived benefits during interaction. The study extends the application of Privacy Calculus Theory by identifying digital literacy as an important antecedent of user evaluation and highlighting the central role of trust in conversational AI contexts. Practically, the findings suggest that developers should enhance transparency, trust-building mechanisms, and privacy communication to improve user evaluations. Future research may further examine additional system-related factors and diverse user groups to validate and extend the proposed framework.

Author Biographies

  • Guan Xian, School of Arts, Universiti Sains Malaysia

    Guan Xian is a PhD student at the School of Arts, Universiti Sains Malaysia, 11800 USM, Penang, Malaysia.

  • Norfarizah Mohd Bakhi, School of Arts, Universiti Sains Malaysia

    Dr. Norfarizah Mohd Bakhir is a senior lecturer in the New Media Design and Technology Department at the School of Art, Universiti Sains Malaysia. Her research areas include New Media Design, Visual Communication, Educational Technology, Film and Media, and Digital Heritage. She is currently supervising PhD students in these fields.

References

Al-Sharafi, M. A., Al-Emran, M., Arpaci, I., Iahad, N. A., AlQudah, A. A., Iranmanesh, M., & Al-Qaysi, N. (2023). Generation Z use of artificial intelligence products and its impact on environmental sustainability: A cross-cultural comparison. Computers in Human Behavior, 143, 107708. https://doi.org/10.1016/j.chb.2023.107708

Anderson, A. A., Brossard, D., Scheufele, D. A., Xenos, M. A., & Ladwig, P. (2014). The “nasty effect:” Online incivility and risk perceptions of emerging technologies. Journal of Computer-Mediated Communication, 19(3), 373–387. https://doi.org/10.1111/jcc4.12009

Bach, T. A., Khan, A., Hallock, H., Beltrão, G., & Sousa, S. (2024). A systematic literature review of user trust in AI-enabled systems: An HCI perspective. International Journal of Human– Computer Interaction, 40(5), 1251–1266. https://doi.org/10.1080/10447318.2022.2138826

Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023). A mixed-methods approach to understanding user trust after voice assistant failures. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Article 391, 1–16). https://doi.org/10.1145/3544548.3581152

Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology, 39(5), 775–786. https://doi.org/10.1111/j.1467-8535.2007.00793.x

Bhandari, U., Chang, K., & Neben, T. (2019). Understanding the impact of perceived visual aesthetics on user evaluations: An emotional perspective. Information & Management, 56(1), 85–93. https://doi.org/10.1016/j.im.2018.07.003

Bhoj, N., & Bhadoria, R. S. (2022). Time-series based prediction for energy consumption of smart home data using hybrid convolution-recurrent neural network. Telematics and Informatics, 75, 101907. https://doi.org/10.1016/j.tele.2022.101907

Bozkurt, A., & Sharma, R. C. (2024). Trust, credibility and transparency in human-AI interaction: Why we need explainable and trustworthy AI and why we need it now. Asian Journal of Distance Education, 19(2), i–ix. https://doi.org/10.5281/zenodo.14599168

Caroline, A., Coun, M. J., Gunawan, A., & Stoffers, J. (2025). A systematic literature review on digital literacy, employability, and innovative work behavior: Emphasizing the contextual approaches in HRM research. Frontiers in Psychology, 15, 1448555. https://doi.org/10.3389/fpsyg.2024.1448555

Chan, C. K. Y., & Lee, K. K. (2023). The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and millennial generation teachers? Smart Learning Environments, 10(1), 60. https://doi.org/10.1186/s40561-023-00269-3

Cyr, D., Head, M., & Larios, H. (2010). Colour appeal in website design within and across cultures: A multi-method evaluation. International Journal of Human-Computer Studies, 68(1–2), 1–21. https://doi.org/10.1016/j.ijhcs.2009.08.005

Dewalska-Opitek, A., Witczak, O., Szostak, A., Dziura, M., & Wroniszewska-Drabek, B. (2024). Generation Z's trust toward artificial intelligence: Attitudes and opinions. European Research Studies Journal, 27(Special Issue 3), 33–52. https://doi.org/10.35808/ersj/3475

Dinev, T., & Hart, P. (2006). An extended privacy calculus model for e-commerce transactions. Information Systems Research, 17(1), 61–80. https://doi.org/10.1287/isre.1060.0080

Djafarova, E., & Foots, S. (2022). Exploring ethical consumption of generation Z: Theory of planned behaviour. Young Consumers, 23(3), 413–431. https://doi.org/10.1108/YC-10-2021-1405

Fox, G., Lynn, T., & Rosati, P. (2022). Enhancing consumer perceptions of privacy and trust: A GDPR label perspective. Information Technology & People, 35(8), 181–204. https://doi.org/10.1108/ITP-09-2021-0706

Ghozali, M. T., Satibi, & Fortwengel, G. (2025). Adoption intentions toward AI-based clinical decision support tools: A Tam study on hospital pharmacists. Journal of Information and Communication Technology, 24(3), 44–69. https://doi.org/10.32890/jict2025.24.3.3

Glassberg, I., Ilan, Y. B., & Zwilling, M. (2025). The key role of design and transparency in enhancing trust in AI-powered digital agents. Journal of Innovation & Knowledge, 10(5), 100770. https://doi.org/10.1016/j.jik.2025.100770

González-Rodríguez, M. R., Díaz-Fernández, M. C., & Gómez, C. P. (2020). Facial-expression recognition: An emergent approach to the measurement of tourist satisfaction through emotions. Telematics and Informatics, 51, 101404. https://doi.org/10.1016/j.tele.2020.101404

Hair, J. F., Jr., Hult, G. T. M., Ringle, C. M., Sarstedt, M., Danks, N. P., & Ray, S. (2021). Partial least squares structural equation modeling (PLS-SEM) using R: A workbook. Springer. https://doi.org/10.1007/978-3-030-80519-7

Harris-Watson, A. M., Larson, L. E., Lauharatanahirun, N., DeChurch, L. A., & Contractor, N. S. (2023). Social perception in human-AI teams: Warmth and competence predict receptivity to AI teammates. Computers in Human Behavior, 145, 107765. https://doi.org/10.1016/j.chb.2023.107765

Hassanein, K., & Head, M. (2007). Manipulating perceived social presence through the web interface and its impact on attitude towards online shopping. International Journal of Human-Computer Studies, 65(8), 689–708. https://doi.org/10.1016/j.ijhcs.2006.11.018

Hassenzahl, M. (2007). The hedonic/pragmatic model of user experience. In E. Law, A. Vermeeren, M. Hassenzahl, & A. Blythe (Eds.), Towards a UX manifesto (pp. 10–14). COST Action 294: MAUSE.

Hassenzahl, M., & Ullrich, D. (2007). To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers, 19(4), 429–437. https://doi.org/10.1016/j.intcom.2007.05.001

He, T., Huang, C., Li, M., Zhou, Y., & Li, S. (2020). Social participation of the elderly in China: The roles of conventional media, digital access and social media engagement. Telematics and Informatics, 48, 101347. https://doi.org/10.1016/j.tele.2020.101347

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/10.1007/s11747-014-0403-8

Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

Hwang, Y. (2023). When makers meet the metaverse: Effects of creating NFT metaverse exhibition in maker education. Computers & Education, 194, 104693. https://doi.org/10.1016/j.compedu.2022.104693

Joshi, H. (2025). Integrating trust and satisfaction into the UTAUT model to predict chatbot adoption– A comparison between Gen-Z and millennials. International Journal of Information Management Data Insights, 5(1), 100332. https://doi.org/10.1016/j.jjimei.2025.100332

Kim, Y., Kim, S. H., Peterson, R. A., & Choi, J. (2023). Privacy concern and its consequences: A meta-analysis. Technological Forecasting and Social Change, 196, 122789. https://doi.org/10.1016/j.techfore.2023.122789

Kirschner, P. A., & De Bruyckere, P. (2017). The myths of the digital native and the multitasker. Teaching and Teacher Education, 67, 135–142. https://doi.org/10.1016/j.tate.2017.06.001

Koester, N., Cichy, P., Antons, D., & Salge, T. O. (2022). Perceived privacy risk in the Internet of Things: Determinants, consequences, and contingencies in the case of connected cars. Electronic Markets, 32(4), 2333–2355. https://doi.org/10.1007/s12525-022-00522-6

Lan, M., Liu, H., & Pan, Q. (2025). Unpacking the digital literacy–self-regulated learning nexus: A systematic review and a three-level meta-analysis. Educational Research Review, 100713. https://doi.org/10.1016/j.edurev.2025.100713

Langer, M., & Landers, R. N. (2021). The future of artificial intelligence at work: A review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers. Computers in Human Behavior, 123, 106878. https://doi.org/10.1016/j.chb.2021.106878

Leschanowsky, A., Rech, S., Popp, B., & Bäckström, T. (2024). Evaluating privacy, security, and trust perceptions in conversational AI: A systematic review. Computers in Human Behavior, 159, 108344. https://doi.org/10.1016/j.chb.2024.108344

Liao, Q. V., & Sundar, S. S. (2022). Designing for responsible trust in AI systems: A communication perspective. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1257–1268. https://doi.org/10.1145/3531146.3533182

Malhotra, N. K., Kim, S. S., & Agarwal, J. (2004). Internet users’ information privacy concerns (IUIPC): The construct, the scale, and a causal model. Information Systems Research, 15(4), 336–355. https://doi.org/10.1287/isre.1040.0032

Männasoo, K., Pareliussen, J. K., & Saia, A. (2023). Digital capacity and employment outcomes: Microdata evidence from pre-and post-COVID-19 Europe. Telematics and Informatics, 83, 102024. https://doi.org/10.1016/j.tele.2023.102024

Mannell, K. (2020). Plural and porous: Reconceptualizing the boundaries of mobile messaging group chats. Journal of Computer-Mediated Communication, 25(4), 274–290. https://doi.org/10.1093/jcmc/zmaa007

Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734. https://doi.org/10.2307/258792

McKee, K. M., Dahl, A. J., & Peltier, J. W. (2024). Gen Z’s personalization paradoxes: A privacy calculus examination of digital personalization and brand behaviors. Journal of Consumer Behaviour, 23(2), 405–422. https://doi.org/10.1002/cb.2199

McKnight, D. H., Choudhury, V., & Kacmar, C. (2002). Developing and validating trust measures for e-commerce: An integrative typology. Information Systems Research, 13(3), 334–359. https://doi.org/10.1287/isre.13.3.334.81

Midha, V., & Nandedkar, A. (2012). Impact of similarity between avatar and their users on their perceived identifiability: Evidence from virtual teams in Second Life platform. Computers in Human Behavior, 28(3), 929–932. https://doi.org/10.1016/j.chb.2011.12.013

Moll, J., Huang, Y., & Sallnäs, E.-L. (2010). Audio makes a difference in haptic collaborative virtual environments. Interacting with Computers, 22(6), 544–555. https://doi.org/10.1016/j.intcom.2010.06.001

Ng, S. W. T., & Zhang, R. (2025). Trust in AI-driven chatbots: A systematic review. Telematics and Informatics, 102240. https://doi.org/10.1016/j.tele.2025.102240

Ng, W. (2012). Can we teach digital natives digital literacy? Computers & Education, 59(3), 1065– 1078. https://doi.org/10.1016/j.compedu.2012.04.016

Otten, R., Schrepp, M., & Thomaschewski, J. (2020). Visual clarity as mediator between usability and aesthetics. Proceedings of Mensch und Computer 2020 (pp. 11–15). Association for Computing Machinery. https://doi.org/10.1145/3404983.3409990

Pan, Z., Moore, O. A., Papadimitriou, A., & Zhu, J. (2025). AI literacy and trust: A multi-method study of human-GAI team collaboration. Computers in Human Behavior: Artificial Humans, 100162. https://doi.org/10.1016/j.chbah.2025.100162

Priporas, C.-V., Stylos, N., & Fotiadis, A. K. (2017). Generation Z consumers’ expectations of interactions in smart retailing: A future agenda. Computers in Human Behavior, 77, 374–381. https://doi.org/10.1016/j.chb.2017.01.058

Ramezani Nia, M., & Shokouhyar, S. (2020). Analyzing the effects of visual aesthetic of web pages on users’ responses in online retailing using the VisAWI method. Journal of Research in Interactive Marketing, 14(4), 357–389. https://doi.org/10.1108/JRIM-11-2018-0147

Rheu, M., Shin, J. Y., Peng, W., & Huh-Yoo, J. (2021). Systematic review: Trust-building factors and implications for conversational agent design. International Journal of Human–Computer Interaction, 37(1), 81–96. https://doi.org/10.1080/10447318.2020.1807710

Riedl, R. (2022). Is trust in artificial intelligence systems related to user personality? Review of empirical evidence and future research directions. Electronic Markets, 32(4), 2021–2051. https://doi.org/10.1007/s12525-022-00594-4

Rossini, P., Stromer-Galley, J., Baptista, E. A., & Veiga de Oliveira, V. (2021). Dysfunctional information sharing on WhatsApp and Facebook: The role of political talk, cross-cutting exposure and social corrections. New Media & Society, 23(8), 2430–2451. https://doi.org/10.1177/1461444820928059

Smahel, D., Machackova, H., Mascheroni, G., Dedkova, L., Staksrud, E., Ólafsson, K., Livingstone, S., & Hasebrink, U. (2020). EU Kids Online 2020: Survey results from 19 countries. EU Kids Online. https://doi.org/10.21953/lse.47fdeqj01ofo

Soliman, M., Ali, R. A., Mahmud, I., & Noipom, T. (2025). Unlocking AI-powered tools adoption among university students: A fuzzy-set approach. Journal of Information and Communication Technology, 24(1), 1–28. https://doi.org/10.32890/jict2025.24.1.1

Susilawati, S., Chakim, A., Hambali, M., Islamy, M. I., & Rahmaniah, A. (2021). The urgency of digital literacy for Generation Z in improving learning of Islamic religious education. Library Philosophy and Practice (e-journal), Article 5143. https://digitalcommons.unl.edu/libphilprac/5143

Tinmaz, H., Lee, Y.-T., Fanea-Ivanovici, M., & Baber, H. (2022). A systematic review on digital literacy. Smart Learning Environments, 9(1), 21. https://doi.org/10.1186/s40561-022-00204-y

Tractinsky, N., Katz, A. S., & Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13(2), 127–145. https://doi.org/10.1016/S0953-5438(00)00031-X

Van Deursen, A. J., & Van Dijk, J. A. (2014). The digital divide shifts to differences in usage. New Media & Society, 16(3), 507–526. https://doi.org/10.1177/1461444813487959

Van Laar, E., Van Deursen, A. J., Van Dijk, J. A., & De Haan, J. (2020). Determinants of 21st-century skills and 21st-century digital skills for workers: A systematic literature review. Sage Open, 10(1), 2158244019900176. https://doi.org/10.1177/2158244019900176

Xiao, L., Pyng, H. S., Ayub, A. F. M., Zhu, Z., Gao, J., & Qing, Z. (2025). University students’ usage of generative artificial intelligence for sustainability: A cross-sectional survey from China. Sustainability, 17(8), 3541. https://doi.org/10.3390/su17083541

Xing, B., Si, H., Chen, J., Ye, M., & Shi, L. (2021). Computational model for predicting user aesthetic preference for GUI using DCNNs. CCF Transactions on Pervasive Computing and Interaction, 3(2), 147–169. https://doi.org/10.1007/s42486-021-00064-4

Yang, R., & Wibowo, S. (2022). User trust in artificial intelligence: A comprehensive conceptual framework. Electronic Markets, 32(4), 2053–2077. https://doi.org/10.1007/s12525-022-00592-6

Yu, S., Carroll, F., & Bentley, B. L. (2026). Rethinking AI literacy education in higher education: Bridging risk perception and responsible adoption. Social Sciences & Humanities Open, 13, Article 102751. https://doi.org/10.1016/j.ssaho.2026.102751

Zhang, R., & Wang, J. (2025). Perceptions, adoption intentions, and impacts of generative AI among Chinese university students. Current Psychology, 44(11), 11276–11295. https://doi.org/10.1007/s12144-025-07928-3

Zhou, S., & Liu, Y. (2023). Effects of perceived privacy risk and disclosure benefits on the online privacy protection behaviors among Chinese teens. Sustainability, 15(2), 1657. https://doi.org/10.3390/su15021657

Downloads

Published

31-10-2026

Issue

Vol. 25 No. 4 (2026): Journal of Information and Communication Technology (JICT) Vol.25, No.4, October 2026

Section

Articles

License

Copyright (c) 2026 Journal of Information and Communication Technology

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

Xian, G., & Mohd Bakhi, N. (2026). User Evaluation of Conversational AI Systems: The Roles of Digital Literacy, Trust, and Perceived Privacy Risk. Journal of Information and Communication Technology, 25(4), 53-72. https://doi.org/10.32890/jict2026.25.4.3

Research impact

Harvested 2026-10-18
0 citations recorded so far

Counts differ between services because each indexes a different body of literature. None of them is the whole picture.

Identifiers DOI 10.32890/jict2026.25.4.3