Improving Visual Style Classification in Digital Games Using Intercoder Reliability Assessment


  • Jazmi Izwan Jamal Future Creative School, Faculty of Animation and Multimedia, Akademi Seni Budaya dan Warisan Kebangsaan (ASWARA), Malaysia
  • Mohd Hafizuddin Mohd Yusof School of Creative Media, Bahrain Polytechnic, Bahrain
  • Kok Yoong Lim Faculty of Creative Multimedia, Multimedia University, Malaysia
  • Jamia Azdina Jamal Centre for Drug and Herbal Development, Faculty of Pharmacy, Universiti Kebangsaan Malaysia, Malaysia



visual style classification, digital game, Fleiss' kappa model, intercoder reliability, machine-learning


The digital gaming community appreciates visual style information in digital games as it facilitates information seeking. Nevertheless, learned scholars have discovered that the digital game visual style classification is inconsistent and easily modified, potentially limiting
the information and leading to inaccurate visual terminologies during information discovery. Therefore, this cross-sectional study was
performed to assess multiple visual style classification terms and their definitions among Malaysian game developers using the closed
card sorting exercise. A total of seven professional game developers participated in an online survey that comprised thirty-five digital game case studies using a card sorting technique. They were asked to classify nineteen visual style classification terms, including psychedelic, text, illusionism, photorealism, televisualism, handicraft, caricature, celshaded, comic book (anime), watercolour, Lego, minimalism, pixel art, silhouette, bright, dark, maplike, colourful, and black and white. The Fleiss’ kappa intercoder reliability assessment was performed to measure the coders’ agreement on visual style classification, followed by the think-aloud protocol descriptive analysis to gather assessment insights into the visual style descriptions. The intercoder reliability test achieved a significantly moderate agreement based on the results. The professional game developers agreed on eighteen visual styles
and rejected the bright visual style classification due to its overlapping description with the colourful visual style. The definition of ten visual style classifications was improved from the existing Video Game Metadata Schema (VGMS) description, contributing to the digital game’s coherence and consistency. This improvement will enhance visual style classification information for machine-learning-based recommendation systems for digital game distribution platforms and digital archiving.


Metrics Loading ...

Additional Files



How to Cite

Jamal, J. I., Mohd Yusof, M. H., Lim, K. Y., & Jamal, J. A. (2023). Improving Visual Style Classification in Digital Games Using Intercoder Reliability Assessment. Journal of Information and Communication Technology, 22(2), 283–308.