ODC - Orthogonal Defect Classification

Subjective aspects of Growth Curves

Before combining ODC with growth curves, it is important that we discuss, in general, some of the qualitative types of growth curves. Figure 2 shows a typical reliability growth curve with the cumulative number of defects on the ordinate and calendar time on the abscissa. This is one of the fairly standard representations, which will be the one discussed in this paper. There are, however, several variations of these: The ordinate may represent failure rate, and sometimes failure density. The abscissa may represent calendar time, execution time, test cases run, percent of test cases, etc. However, for this paper, we restrict our attention to growth curves with the cumulative number of defects on the ordinate and calendar time on the abscissa. The ideas that are proposed may map into other representations, however, that discussion is not the focus of this paper.

There is a subtlety that is often not discussed in growth models - i.e. what time period is captured in the abscissa of such a representation? We believe it is a very significant part of the discussion, that is often overlooked. It is important for several reasons - the simplest being that the units of measure that are used for the abscissa should have some consistency across all the phases of software development that it spans. For instance, if the abscissa spans, code inspection, unit test, function test and system test, one has to question if days make sense all through that span of the process. If not, there should be some normalization done, so that the measure of reliability on the ordinate makes sense. This question gets deeper, when considering that the work is done by a team and the units should be normalized to reflect either the effort by the team or the coverage of the product. We raise this point to bring to attention that this process is fairly complicated and the answers are not uniformly available. In this paper, we have chosen the span of time from the beginning of the function test until the product is shipped. Based on communication with developers, we believe that this choice provides a reasonably consistent time scale.

The growth curve in Figure 2 is meant to show a span of development periods that begin around the time of the function/component test (FCT), continued into the system test and possibly the early part of the field introduction. The part of the curve which has an accelerated defect detection rate is usually where the function test is ramping up. During FCT not all the code is integrated, but there is probably a substantial amount of parallel testing of the components with the test cases developed, alongside the product. At some time, when most of the test cases have been run, the product is integrated and the system test is begun that starts to stress the code. The ideal time to release the product is after the knee in the curve, which reduces the exposure of defects being found in the field. In practice, several variances occur. Products are not shipped at the knee, as would be ideally desired. The knee can occur in the field. When one does ship just after the knee, there is often another knee that develops a few months out into the field.

  
Figure 2: Typical Growth Curve

For the discussion in this paper, we identify one key point on the growth curve, projected on the time line, which is used to compare different growth curves. We find it useful, from practice, to identify the point, when 90% of the predicted total number of defects are found. In the figure, this is annotated as , and referred to, in this paper, as the point. It is intended to approximately represent the knee of the curve. It may or may not be visible depending on how far the growth curve has progressed. When visible, it provides a useful milestone in the development cycle to make comparative assessments.