Two-Phase Defect Detection Using Clustering and Classification Methods
Abstract
Autonomous fault management of network and distributed systems is a challenging research problem and attracts many research activities. Solving this problem heavily depends on expertise knowledge and supporting tools for monitoring and detecting defects automatically. Recent research activities have focused on machine learning techniques that scrutinize system output data for mining abnormal events and detecting defects. This paper proposes a two-phase defect detection for network and distributed systems using log messages clustering and classification. The approach takes advantage of K-means clustering method to obtain abnormal messages and random forest method to detect the relationship of the abnormal messages and the existing defects. Several experiments have evaluated the performance of this approach using the log message data of Hadoop Distributed File System (HDFS) and the bug report data of Bug Tracking System (BTS). Evaluation results have disclosed some remarks with lessons learned.
Published
2022-06-14
Section
Regular articles
An author's submission implies that the manuscript has not been published previously, and is not currently submitted for publication elsewhere. Submission also implies that the Corresponding Author has consent of all authors (the Authors). Upon acceptance for publication transfer of copyright will be made to the Publisher of REV-JEC, who guarantees that full content of the published article is freely distributed on the Journal's website. The copyright transfer gives the Publisher of REV-JEC full authority to resolve any complaints of misuse or abuse (such as infringement or plagiarism) of the published article. The Authors have the freedom to redistribute and reuse the published article in any medium or format for any purpose, provided the original published article is properly cited. An article submission implies author agreement with this policy.
