Improving Visual Style Classification in Digital Games Using Intercoder Reliability Assessment
DOI:
https://doi.org/10.32890/jict2023.22.2.6Keywords:
visual style classification, digital game, Fleiss' kappa model, intercoder reliability, machine-learningAbstract
The digital gaming community appreciates visual style information in digital games as it facilitates information seeking. Nevertheless, learned scholars have discovered that the digital game visual style classification is inconsistent and easily modified, potentially limiting
the information and leading to inaccurate visual terminologies during information discovery. Therefore, this cross-sectional study was
performed to assess multiple visual style classification terms and their definitions among Malaysian game developers using the closed
card sorting exercise. A total of seven professional game developers participated in an online survey that comprised thirty-five digital game case studies using a card sorting technique. They were asked to classify nineteen visual style classification terms, including psychedelic, text, illusionism, photorealism, televisualism, handicraft, caricature, celshaded, comic book (anime), watercolour, Lego, minimalism, pixel art, silhouette, bright, dark, maplike, colourful, and black and white. The Fleiss’ kappa intercoder reliability assessment was performed to measure the coders’ agreement on visual style classification, followed by the think-aloud protocol descriptive analysis to gather assessment insights into the visual style descriptions. The intercoder reliability test achieved a significantly moderate agreement based on the results. The professional game developers agreed on eighteen visual styles
and rejected the bright visual style classification due to its overlapping description with the colourful visual style. The definition of ten visual style classifications was improved from the existing Video Game Metadata Schema (VGMS) description, contributing to the digital game’s coherence and consistency. This improvement will enhance visual style classification information for machine-learning-based recommendation systems for digital game distribution platforms and digital archiving.
References
Cho, H., Donovan, A., & Lee, J. H. (2018). Art in an algorithm: A taxonomy for describing video game visual styles. Journal of Journal of ICT, 22, No. 2 (April) 2023, pp: 283– the Association for Information Science and Technology, 69(5), 633–646. https://doi.org/10.1002/asi.23988
Cho, H., Pham, M. T. N., Leonard, K. N., & Urban, A. C. (2022). A systematic literature review on image information needs and behaviors. Journal of Documentation, 78(2), 207–227. https://doi.org/10.1108/JD-10-2020-0172
Clarke, R. I., Jett, J., Sacchi, S., & Lee, J. H. (2016). A conceptual model for video games and interactive media recommended citation. School of Information Studies: Faculty Scholarship, 171. https://surface.syr.edu/istpub
Clarke, R. I., Lee, J. H., & Rossi, S. (2015). A qualitative investigation of users’ video game information. School of Information Studies - Faculty Scholarship, 166. https://surface.syr.edu/istpub/166
Connor, M., & Lawrence, A. B. (2017). Understanding adolescents’ categorisation of animal species. Animals, 7(9), 65. https://doi.org/10.3390/ani7090065
Donovan, A., Cho, H., Magnifico, C., & Lee, J. H. (2013, July). Pretty as a pixel: Issues and challenges in developing a controlled vocabulary for video game visual styles. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (pp. 413–414). IEEE. https://doi.org/10.1145/2467696.2467747
Enderlein, G. (1988). Fleiss, J. L.: The Design and Analysis of Clinical Experiments. Wiley, New York – Chichester – Brislane – Toronto – Singapore 1986, 432 S., £38.35. Biometrical Journal, 30(3), 304. https://doi.org/https://doi.org/10.1002/ bimj.4710300308
Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions. Wiley & Sons.
Fleiss, J. L., Levin, B., & Paik, M. C. (2004). The measurement of interrater agreement. In Statistical Methods for Rates and Proportions (pp. 598–626). https://doi.org/10.1002/0471445428.ch18
Fleiss, J. L., Nee, J. C. M., & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974–977. https://psycnet.apa. org/buy/1979-32706-001
Halpern, J. (2018). Developing 2D games with unity. New York City: Apress.
Heljakka, K. (2018). Bel far niente photography as productive play in creative cultures of the 21st century. In Fotografia e culture visuali del XXI secolo (pp. 473–478). Journal of ICT, 22, No. 2 (April) 2023, pp: 283–
Jett, J., Sacchi, S., Lee, J. H., & Clarke, R. I. (2016). A conceptual model for video games and interactive media. Journal of the Association for Information Science and Technology, 67(3), 505–517. https://doi.org/10.1002/asi.23409
Keating, S., Lee, W.-C., Windleharth, T., & Lee, J. H. (2017, January). The style of Tetris is… possibly Tetris: Creative professionals description of video game visual styles. In Proceedings of the 50th Hawaii International Conference on System Sciences (2017) (pp. 2046–2055). https://doi.org/10.24251/ hicss.2017.248
Khoo, F. S., Teh, P. L., & Ooi, P. B. (2017). Consistency of online consumers’ perceptions of posted comments: An analysis of tripadvisor reviews. Journal of Information and Communication Technology, 16(2), 374–393. http://www.jict.uum.edu.my/ images/pdf4/vol16no22017/374-393.pdf
Konstantinidis, M., Le, L. W., & Gao, X. (2022). An empirical comparative assessment of inter-rater agreement of binary outcomes and multiple raters. Symmetry, 14(2), 262. https://doi.org/10.3390/sym14020262
Krupke, D., Zhang, J., & Steinicke, F. (2019). IMPAct: A holistic framework for mixed reality robotic user interface classification and design. Multimodal Technologies and Interaction, 3(2), 25. https://doi.org/10.3390/mti3020025
Lee, J. H., Cho, H., Fox, V., & Perti, A. (2013, July). User-centered approach in creating a metadata schema for video games and interactive media. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (pp. 229–238). IEEE. https://doi.org/10.1145/2467696.2467702
Lee, J. H., Clarke, R. I., & Kim, Y.-S. (2015a). Video game information needs and game organization: Differences by sex and age. Information Research, 20(3). http://informationr.net/ir/20-3/ paper683.html#.YPeCxugzbcQ
Lee, J. H., Clarke, R. I., & Perti, A. (2015b). Empirical evaluation of metadata for video games and interactive media. Journal of the Association for Information Science and Technology, 66(12), 2609–2625. https://doi.org/10.1002/asi.23357
Lee, J. H., Hong, S., Cho, H., & Kim, Y. S. (2015, April). VIZMO game browser: Accessing video games by visual style and mood. In Conference on Human Factors in Computing Systems - Proceedings, 2015 (pp. 149–152). https://doi.org/10.1145/2702123.2702264 Journal of ICT, 22, No. 2 (April) 2023, pp: 283–
Lee, J. H., Perti, A., Cho, H., Donovan, A., Magnifico, C., & Pittman, K. (2014). University of Washington / Seattle Interactive Media Museum video game metadata schema: Controlled vocabulary for visual style. Version 1.5. University of Washington. http://gamer.ischool.uw.edu/wp-content/uploads/2018/04/ VGMSVersion4.0_20180824.pdf
Lee, J. H., Perti, A., Cho, H., Donovan, A., Magnifico, C., & Pittman, K. (2015). Video game metadata schema : Controlled vocabulary for visual style. http://gamer.ischool.uw.edu/official_release/
Lee, J. H., Perti, A., Clarke, R. I., Windleharth, T. W., & Schmalz, M. (2017). University of Washington / Seattle Interactive Media Museum video game metadata schema version 4.0. University of Washington. http://gamer.ischool.uw.edu/wp-content/ uploads/2018/04/VGMSVersion4.0_20180824.pdf
Lee, J. H., Tennis, J. T., & Clarke, R. I. (2012). Domain analysis for a video game metadata schema: Issues and challenges. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7489 LNCS (pp. 280–285). https://doi.org/10.1007/978-3-642-33290-6_30
Mäyrä, F., & Järvinen, A. (2002, June). Gran Stylissimo: The audiovisual elements and styles in computer and video games. In Proceedings of Computer Games and Digital Cultures (pp. 113–128). http://www.digra.org/wp-content/uploads/digital-library/05164.35393.pdf
Mumcu, S., & Yılmaz, S. (2018). Anime landscapes as a tool for analyzing the human–environment relationship: Hayao Miyazaki Films. Arts, 7(2), 16. https://doi.org/10.3390/ arts7020016
Rossi, S., Lee, J. H., & Clarke, R. I. (2014, September). Mood metadata for video games and interactive media. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (pp. 475–476). IEEE. https://doi.org/10.1109/JCDL.2014.6970232
Schwabe, S. A., & Caldwell, S. (2022). Can anteroposterior skeletal pattern be determined from a silhouetted profile photograph? A cross-sectional study. Journal of Orthodontics, 49(1), 48–55. https://doi.org/10.1177/14653125211028862
Skulmowski, A., Nebel, S., Remmele, M., & Rey, G. D. (2022). Is a preference for realism really naive after all? A cognitive model of learning with realistic visualizations. Educational Psychology Review, 34(2), 649–675. https://doi.org/10.1007/ s10648-021-09638-1 Journal of ICT, 22, No. 2 (April) 2023, pp: 283–
Su, Y. (2018). The application of 3D technology in video games. Journal of Physics: Conference Series, 1087(6), 62024. https://doi.org/10.1088/1742-6596/1087/6/062024
Tennis, J. T., Lee, J. H., Clarke, R. I., & Carpenter, M. (2015). Developing a video game metadata schema for the Seattle Interactive Media Museum. University of Washington. https://digital.lib.washington.edu:443/researchworks/ handle/1773/33392
Vert, S., Andone, D., Ternauciuc, A., Mihaescu, V., Rotaru, O., Mocofan, M., Orhei, C., & Vasiu, R. (2021). User evaluation of a multi-platform digital storytelling concept for cultural heritage. Mathematics, 9(21), 2678. https://doi.org/10.3390/ math9212678
Viera, A. J., & Garret, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37(5), 360–363. http://www1.cs.columbia.edu/~julia/courses/CS6998/ Interrater_agreement.Kappa_statistic.pdf
Warrens, M. J. (2010). Inequalities between multi-rater kappas. Advances in Data Analysis and Classification, 4(4), 271–286. https://doi.org/10.1007/s11634-010-0073-4
Welhouse, Z., Lee, J. H., & Bancroft, J. (2015). “What am I fighting for?”: Creating a controlled vocabulary for video game plot metadata. Cataloging and Classification Quarterly, 53(2), 157–189. https://doi.org/10.1080/01639374.2014.963776
Published
Issue
Section
License
Copyright (c) 2023 Journal of Information and Communication Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Research impact
Harvested 2026-09-06Counts differ between services because each indexes a different body of literature. None of them is the whole picture.
2002 - 2020
