ODC - Orthogonal Defect Classification

Summary

Orthogonal Defect Classification makes a fundamental improvement in the technology for in-process measurement for the software development processes. This opens up new opportunity for developing models, and techniques for fast feedback to the developer, addressing a key challenge that has been nagging the community for years.

At one end of the spectrum, research in defect modeling, that focused on reliability prediction, treats all defects as homogeneous. At the other end of the spectrum, causal analysis provides qualitative feedback on the process at a very small granularity. The middle ground did not develop systematic mechanisms for feedback due to the lack of fundamental cause-effect relationship extractable from the process. This work is built on some fundamental discoveries based on semantic extraction via classification that is carefully constructed to address the issue. Orthogonal Defect Classification provides a basic capability to extract signatures from defects and infer the health of the development process. The classification is to be based on what was known about the defect such as its defect type or trigger and not on opinion such as where it should have been found. The choice of the classes in an attribute should satisfy the stated necessity and sufficient conditions so that they eventually point to the part of the process that requires attention.

The design of the defect type attribute measures the progress of a product through the process. Defect type identifies what is corrected and can be associated with the different stages of the process. Thus, a set of defects from different stages in the process, classified according to an orthogonal set of attributes, should bear the signature of this stage in its distribution. Moreover, changes in the distribution can meter the progress of the product through the process. The departure from the expected distribution alerts us by pointing to the stage of the process that requires attention. Thus, the defect type provides feedback on the development process.

The design of the defect trigger attribute (not covered in this short overview paper, but available in the original reference) provides a measure of the effectiveness of a verification stage. Defect triggers capture the circumstance that allowed the defect to surface. The information that yields the trigger measures aspects of completeness of a verification stage. The verification stages could be the testing of code or the inspection and review of a design. These data can eventually provide feedback on the verification process. Taken together with the defect type, the cross-product of defect type and trigger provides information that can estimate the effectiveness of the process.

Our experience with ODC, indicates that it can provide fast feedback to developers. Currently, two stage data is used for trend analysis to yield feedback. It is envisioned that as pilots evolve, the measurements can yield calibration.

Developers find this a useful method, giving them insight they did not have before. It also provides a reasonable level of quantification to help make better management decisions to significantly impact cost and opportunity.