ODC - Orthogonal Defect Classification

Introduction

In recent years the emphasis on software quality has increased due to forces from several sectors of the computer industry. Software is one of the slowest processes in enabling new business opportunities, solutions, or dimensions of computing, and is a significant part of total cost. It has also been found that the dominant cause of system outage has shifted to software given that it has not kept pace, in the past few years, with improvements in hardware or maintenance [1]. Thus, there is a resurgence of research in software topics such as Reliability, Engineering, Measurement, etc. [2, 3]. However, the area of software quality measurements and quantification is beset with undue complexity and has, in some ways, advanced away from the developer. In an area where the processes are so amorphous, the tangibles required for measurement and modeling are few. With the result academic pursuits that can't be confined to the limitations of practice evolved and became distanced from the developer. In this area, the need to derive tractable measurements that are reasonable to undertake and intuitively plausible cannot be understated. Measurement without an underlying theme can leave the experimentalist, the theorist and the practitioner very confused.

The goal of this paper is to develop reasonable measurements. It does this by examining the fundamental properties of such measurements and then deriving the rationale for analysis and models. To put the domain of software in-process measurements and analysis in perspective, let us examine two extremes of the spectrum; statistical defect models and qualitative causal analysis.

Statistical Defect Models

The goal of statistical defect modeling, which includes what is commonly referred to as Software Reliability Growth, has been to predict the reliability of a software product. Typically, this may be measured in terms of the number of defects remaining in the field, the failure rate of the product, the short term defect detection rate, etc. [4, 5, 3]. Although this may provide a good report card, it often occurs so late in the development cycle that it is of little value to the developer. Ideally, a developer would like to get feedback during the process.

Causal Analysis

The goal of causal analysis is to identify the root cause of defects and initiate actions so that the source of defects is eliminated. To do so, defects are analyzed, one at a time, by a team that is knowledgeable in the area. The analysis is qualitative and only limited by the range of human investigative capabilities. To sustain such pursuit and provide a focus for the activity a process description has been found useful. At IBM, the Defect Prevention Process [6] and similar efforts elsewhere have found causal analysis to be very effective in reducing the number of errors committed in a software project. The qualitative analysis provides feedback to developers that eventually improves both the quality and the productivity of the software organization [7].

Defect Prevention can provide feedback to developers at any stage of their software development process. However, the resources required to administer this method are significant, although the rewards have proven to be well worth it. Moreover, given the qualitative nature of the analysis, the method does not lend itself well to measurement and quantitative analysis. Consequently, Defect Prevention, though not a part of the engineering process control model, could eventually work in conjunction with it.

The Gap

Between the two extremes of the spectrum - quantitative statistical defect models and qualitative causal analysis, is a wide gap. This gap is characterized by a lack of good measurement methods that are meaningful to the developer and that can exploit good engineering methods. At one end, the traditional mathematical modeling efforts tend to step back from the details of the process and to approximate defect detection by statistical processes [8, 9]. When calibrated, some of these methods are shown to be quite accurate. However, they do not provide timely feedback to the developer in terms of available process controls. At the other end, the causal analysis mechanism is qualitative and labor intensive; it can provide feedback on each individual defect. However, in a large development effort it is akin to studying the ocean floor with a microscope. It does not naturally evolve into abstractions and aggregations that can feed into engineering mechanisms to be used for overall process control.

It is not as though there is no work done between these two extremes, indeed there is a myriad of reported research and industry attempts to quantify the parameters of the software development process with ``metrics'' [10, 11]. Many of these attempts are isolated and suffer from the absence of an overall methodology and a well defined framework for incrementally improving the state of the art. Some efforts have been more successful than others. For example, the relationship between the defects that occur during software development and the complexity of a software product have been discussed in [12]. Such information, when compiled over the history of an organization [13], will be useful for planners and designers. There also is no one standard [10] classification system that is in vogue, although there have been efforts in that direction [14].

In summary, although measurements have been extensively used in Software Engineering, it still remains a challenge to turn software development into a measurable and controllable process. Why is this so? Primarily because no process can be modeled as an observable and controllable system unless explicit input-output or cause and effect relationships are established. Furthermore, such causes and effects should be easily measurable. It is inadequate to propose that a collection of measures be used to track a process, with the hope that some subset of these measures will actually explain the process. There should at least be a small subset which is carefully designed based on a good understanding of the mechanisms within the process.

Looking at the history of the modeling literature in software, it is evident that little heed has been paid to the actual cause-effect mechanism, leave alone investigations to establish them. At the other extreme, when cause-effect was recognized, though qualitatively, it was not abstracted to a level from which it could graduate to engineering models. To the best of the authors knowledge, in the world of in-process measurements, and until recently, there has been no systematic study to establish the existence of measurable cause and effect relationships in a software development process underway. Without that insight, and a rational basis, it is hard to argue that any one measurement scheme or model is better than another.

The Bridge

This paper presents Orthogonal Defect Classification (ODC), a technique that bridges the gap between statistical defect models and causal analysis. It brings a scientific approach to measurements in a difficult area that otherwise can easily become adhoc. It also provides a firm footing from which classes of models and analytical techniques can be systematically derived. The goal is to provide an in-process measurement paradigm to extracting key information from defects and enable the metering of cause-effect relationships. ODC is inspired by a previous study that identified the existence of measurable cause and effect relationships in a software development process [15]. Specifically, the choice of a set of orthogonal classes, mapped over the space of development or verification, can help developers by providing feedback on the progress of their software development efforts. These data and their properties provide a framework for analysis methods that exploit traditional engineering methods of process control and feedback.

ODC is enabled only if certain fundamental criteria are met. Section 2 of this paper discusses the concept of orthogonality in ODC and the necessary and sufficient conditions for a classification scheme to provide feedback. Section 3 is devoted to defect types, and their use to provide feedback on the development process. Section 4 discusses defect triggers. Triggers provide feedback on the verification process just as defect types provide feedback on the development process. Section 5 assesses the costs involved in implementing ODC and its relationship to causal analysis.

The availability of well designed defect tracking tools, education, and pilot programs to test and develop the technique is critical to the success of implementing ODC. The classification system used for these attributes and new attributes are constantly being evolved. For instance, the paradigm illustrated in this paper could just as well be applied to hardware development, information development or non-defect oriented problems. Several test-pilot programs are currently underway to explore these areas. After a series of test pilots, the classification system is stabilized and put into production. However, we are aware that changes in the classification may be necessary and currently they are handled in similarity with software releases.