Improving Visual Style Classification in Digital Games Using Intercoder Reliability Assessment

Abstract

The digital gaming community appreciates visual style information in digital games as it facilitates information seeking. Nevertheless, learned scholars have discovered that the digital game visual style classification is inconsistent and easily modified, potentially limiting
the information and leading to inaccurate visual terminologies during information discovery. Therefore, this cross-sectional study was
performed to assess multiple visual style classification terms and their definitions among Malaysian game developers using the closed
card sorting exercise. A total of seven professional game developers participated in an online survey that comprised thirty-five digital game case studies using a card sorting technique. They were asked to classify nineteen visual style classification terms, including psychedelic, text, illusionism, photorealism, televisualism, handicraft, caricature, celshaded, comic book (anime), watercolour, Lego, minimalism, pixel art, silhouette, bright, dark, maplike, colourful, and black and white. The Fleiss’ kappa intercoder reliability assessment was performed to measure the coders’ agreement on visual style classification, followed by the think-aloud protocol descriptive analysis to gather assessment insights into the visual style descriptions. The intercoder reliability test achieved a significantly moderate agreement based on the results. The professional game developers agreed on eighteen visual styles
and rejected the bright visual style classification due to its overlapping description with the colourful visual style. The definition of ten visual style classifications was improved from the existing Video Game Metadata Schema (VGMS) description, contributing to the digital game’s coherence and consistency. This improvement will enhance visual style classification information for machine-learning-based recommendation systems for digital game distribution platforms and digital archiving.