Development and User Acceptance Evaluation of a Desktop Web Crawler for Keyword-Based Information Retrieval

Abstract

The growing volume of online information makes it difficult for users to find relevant content without manually examining multiple webpages. Although web crawlers can automate information retrieval, many existing tools require programming knowledge or complex configuration. This study developed and evaluated a Windows-based desktop web crawler for keyword-based information retrieval. The prototype followed a six-phase Agile process: planning, design, development, testing, deployment, and review. The team used Java and JavaFX to develop the desktop application and its graphical user interface, Selenium WebDriver to retrieve dynamically rendered webpages, and Java Database Connectivity to connect the application to a MySQL database. The prototype integrates input validation, webpage retrieval, domain link extraction, HTML content cleaning, rule-based keyword matching, URL filtering, result categorisation, database storage, and retrieval-history searching. We assessed the completed prototype through 11 functional test cases and a user acceptance evaluation involving 30 undergraduate students from Universiti Utara Malaysia. Participants tested the prototype using different publicly accessible URLs and subsequently completed a questionnaire measuring their perceptions of input validation, crawling effectiveness, reliability and stability, result-display accuracy, and overall satisfaction. Ten functional test cases achieved a 100.0% pass rate, while link extraction achieved a 96.7% pass rate because one webpage required human verification. The user acceptance evaluation produced mean scores ranging from 4.00 to 4.70 on a five-point Likert scale. Input validation received the highest mean score (M = 4.70), followed by result-display accuracy (M = 4.43) and crawling effectiveness (M = 4.13). Reliability and stability, as well as overall satisfaction, each obtained a mean score of 4.00. The findings indicate that the prototype’s principal functions operated consistently and received generally favourable evaluations under the tested conditions. Further comparative testing is required to determine whether the crawler reduces search time, manual effort, cognitive load, or information overload.