ODC - Orthogonal Defect Classification

Background on ODC

This paper draws from several ideas of Orthogonal Defect Classification (ODC) and combines them with traditional growth modelling. In this section we will briefly review some of the background on ODC pertinent to the discussion and completeness of the paper. A more detailed discussion on the subject can be found in [1].

ODC originally stemmed from the findings, that different types of defects demonstrate fairly radical differences in their growth models [2]. The study showed that when defects were uniquely categorized by a set of defect types, representing the semantics of the fix, it was possible to relate changes in the shape of the reliability growth curve to defects of a specific type. This was exploited to develop a categorization scheme such that the categorized information from a defect provided measurements on the process. ODC requires that the categorization be orthogonal in the value set for an attribute, so that, as the product moves through the process, the distribution of the values change, explaining the progress of the product.

There is a set of necessary and sufficient conditions that makes a classification, defined by attributes and value selections, ODC. This has to do primarily with its capability to explain the progress of a product through the process, using categorical data, extracted from the defects. Fundamentally, the initial research work demonstrated that semantic information from defects, suitably extracted through classification, could provide insight into the progress of the product, [2]. Next, clever techniques to extract this were developed, providing fairly sensitive instruments to measure progress [1]. ODC is currently practiced in several products spread across 12 IBM locations worldwide. Four attributes - defect type, trigger, impact, and source - are added to the existing data captured by IBM defect tracking tools. These data provide a multidimensional view into the development process, and since the carefully designed classification is architected to provide a measurement, they open numerous opportunities for detailed analysis. Contrasted with root-cause analysis, which tells a developer many details on a few specific problems, ODC provides a view of the total phenomenon occurring over a larger population of defects. For this paper, we will be taking advantage of one of the attributes called defect type. The defect type distribution is designed so that its distribution changes as the product progresses through the process. The signature that the distribution generates, provides a far more sensitive instrument than other purely numeric techniques, to gauge the progress of the product. Since we will be using the value set of the defect type attribute in this paper, we will review some of its details.

A programmer making the correction usually chooses the defect type. The defect type signifies the meaning of the eventual correction. These types are simple in that they should be obvious to a programmer, without much room for confusion. In each case a distinction is made between something missing or something incorrect. A function error is one that affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture, or global data structure(s) and should require a formal design change. Conversely, an assignment error indicates a few lines of code, such as the initialization of control blocks or data structure. Interface corresponds to errors interacting with other components, modules or device drivers via macros, call statements, control blocks or parameter lists. Checking addresses program logic which has failed to properly validate data and values before they are used. Timing/serialization errors are those which are corrected by improved management of shared and real-time resources. Build/package/merge describe errors that occur because of mistakes in library systems, management of changes, or version control. Documentation errors can affect both publications and maintenance notes. Algorithm errors include efficiency or correctness problems that affect the task and can be fixed by (re)implementing an algorithm or local data structure without the need for requesting a design change.

The defect types are chosen to be general enough to apply to any software development, independent of a specific product. Other defect classification schemes exist, see, for example, [6]. The ODC defect types, however, are by design, strongly associated with the project phase, and are therefore, expected to follow different growth trajectories. Their granularity is such that they apply to any phase of the development process, yet can be associated with a few specific phases in the process. The function and algorithm defects could be discovered and fixed anywhere in the verification process, but reflect design aspects of the development effort. Assignment and Checking signify defects that are more likely to be coding issues than any other activity, whereas interface reflects low level design type work.

  
Figure 1: Defect Type Distribution by Phase

Figure 1 is a simplified illustration of the change in defect type distribution as a product moves through different phases. In the design phase, the function errors are most frequent, but their relative contribution drops off in later phases. Assignment errors, reflecting coding issues, increase from design phase to unit test and drop somewhat in the later phases. Interface and timing errors increase peak during the integration test and system test respectively.

In the following sections we will exploit the defect type attribute of the ODC classification on defects, to improve the insight that can be gleaned through growth models. This is done by comparing the relative growth of different types of defects.