Software Defect Defense based on Human Error Mechanisms

Abstract

Software defects are the critical threat to increase life-cycle costs, delay project schedule, reduce the reliability of software systems, and even cause catastrophic disasters. Since the concept of software engineering has been proposed, people have developed many technologies to prevent the introduction of software defects. However, the effects are not optimistic. So far although tremendous resources have been devoted to software testing, defects are still the major threat to the reliability of software systems. The proactive defense against software defects can be a promising philosophy to reduce costs and improve reliability. However, conventional relevant technologies such as defect prediction and defect prevention can hardly prevent the introduction of software defects. It is the time to prompt a thorough reflection on the conventional ways: the conventional technologies trend to focus on the improvement of software process, but ignore the underlying mechanisms that cause software defects. Essentially, programs are the “expression” of human thoughts, while software defects are mainly caused by human cognitive failures. Conventional software engineering technologies are designed to control and improve the process of software production, rather than directly impact on the key factor---programmer’s cognition, thus, they can only influence software quality indirectly. Once we have failed to capture the mechanisms of software defects, we can neither predict them precisely, nor prevent them fundamentally. To address these gaps, this thesis proposes the concept of defect defense based on human error mechanisms. Logically, prediction and prevention should be interconnected, since only when an event can be predicted, it can be prevented. That is to say, prediction normally provides implications for prevention. However, due to the omission of mechanisms, the conventional defect prediction is unable to achieve sufficient accuracy at early stages of software development. Thus, conventional predictions can provide little information for defect prevention. That’s why the conventional defect prediction and prevention are completely irrelevant. In this thesis, bonded by the human error mechanisms, prediction and prevention are integrated, to defend against the introduction of software defects together. The research is first carried out by summarizing the relevant research about program design cognition, with an integrated cognition model of program design constructed. Then integrate the classical theories of human errors with the domain characteristics of programming, a base of human error modes for software defects is developed. Based on the integrated cognition model and human error modes, three approaches are proposed, designed and validated. “Conventional defect prevention (DP) based on the structural taxonomy of root causes” is an improved defect prevention approach in the framework of conventional DP. Conventional DP framework is effective in preventing defects due to process problems. However, it is strongly depended on experts’ experiences and brain storms, which have limit its applications in small companies. Even for companies at high process maturity levels, it is hard to replicate the benefits of conventional DP. A structural taxonomy of root causes is proposed and validated, and the core knowledge required for root cause analysis is solidified in the knowledge base. An application case has been carried out, results show that with the assistance of the taxonomy and knowledge base, the small company at the CMM initial level can implement conventional DP effectively. “Defect Prevention by Improving Software Developers’ Meta-cognitive Ability to Prevent Human Errors” (HEDP) is an approach in the framework that is completely different from conventional DP. This approach is proposed for the reason that, individual cognitive failures are the main cause of software defects, but conventional DP has little power in affecting individual’s cognitive performances. HEDP aims to prevent defects by improving programmers’ awareness and regulation abilities under error-prone situations. HEDP is designed in the framework of meta-cognition, including two stages. The first stage concerns meta-cognitive training on human error knowledge and the second stage aims to build programmers’ experience in meta-cognitive regulation. The knowledge training consists of knowledge about program designing cognition, human error mechanisms, and error prevention strategies. The meta-cognitive regulation experience is built by the reflection in the course of problem solving and self-reviews after the defects are detected. Two application cases are studied, with the self-assessment and defect data collected. Both kinds of results show that, HEDP is effective in improving programmers’ meta-cognitive ability to prevent software defects. Furthermore, HEDP is independent of process maturity, that is to say, all organizations can implement HEDP, no matter at CMM level 5 or level 1. Most important of all, HEDP can be used to guide any programmer pursuing self-improvement in human error prevention, no matter experts or novices. “Software defect prediction based on human error mechanisms”(HEFP) is an new approach to predict the location and format of defects at the early phases of software development, i.e. phases of requirement analysis and design. Such prediction is implemented by human error scenario analysis. A controlled experiment has been designed to validate HEFP and provides empirical evidences for relevant concepts. The results show that, HEFP has predominant advantages in predicting coincident defects. HEFP has precisely predicted the location and format of 88.9% coincident defects, which are committed by 96.5% of the subjects who has committed coincident defects. Meanwhile, what the HEFP predicts are the defects at high risk. Though the number of defects predicted by HEFP only constitutes 30.8% of the total defects, but they are committed by 78.6% subjects who commit any error. In comparison, conventional predictors based on program metrics can only account for 26.8% variance of the total defects, and they can not output the accurate locations and formats of the defects. Results show that HEFP performs much better than prediction models based on program metrics, both in the accuracy and efficiency. Most important of all, HEFP can perform at the early phases of the software development, thus it can provide implications for defect prevention. In summary, the two sets of basic theories and three approaches works together, constituting the comprehensive system to defend against software defects.