ODC - Orthogonal Defect Classification

General Issues

The triggers were grouped according to the three activities, namely, review/inspection, function test, and system test triggers. In all, there were 2,770 faults that were classified and Figure 2 shows the distribution of the three different classes of triggers. Notice that only 720 would be attributed to what we may refer to as system test triggers. The others have been classified into triggers associated with either review/inspection or function test. A paradigm that is cherished by many is that the only escapes into the field, one hopes, occur from system test, since it is the last phase in the process. The data reveals that to be only around 25%. 75% of the faults escape from parts of the process that are associated with elements of design or the standardized function and component test. The advantage of looking at faults by trigger is that one can identify very specific actions to target the categories of faults that come under a trigger.

  
Figure 2: Attributing activities via Triggers

Figures 3, 4, and 5 show distributions of each of these groups of triggers as a function of time. The way the charts are drawn one can see that the different triggers tend to peak at different times after product release. This is, in fact, one of the first results ever demonstrated showing that the operational profile aggregated on a product is very much a function of the time after release. Therefore, early in the life of a product there are a different set of triggers that cause failures whereas later in the life of a product, again the mix of triggers that cause failures change. This is true for each of the three groupings of triggers. Once we notice the ones that are peaking, such as documentation or concurrency in the case of review/inspection triggers it is understandable that those would peak early and not later, as is the case with lateral compatibility or logic flow. However, this level of insight has never before been provided.

The fact that each trigger has a different distribution as a function of time is a very key observation. This knowledge opens the doors to a variety of techniques as well as process changes to deal with the issue. There are tactical methods by which we can control the early life fall-out in the field by going after the triggers that tend to dominate the early fault discovery. Usually after about three months in the field, there is a possibility of providing a service pact to the field in order to cover known faults and potentially avoid future faults. Knowing that certain triggers actually peak later in the product life gives us a tactical opportunity to focus on certain kinds of testings ahead of other kinds of tests. There is usually much debate about the reason why faults escaped into the field. It is often argued that testing is never complete and that escapes into the field are either dominated by design issues or non-design issues. This kind of discussion is usually based on opinions of people rather than quantified evidence as to why a particular fault escaped the development process. The technique we use here is to first identify the trigger, offering a crisp articulation of the reason why a fault was not caught. Then, depending on the trigger attribute, we are able to discern the part of the process that allowed an escape. For instance, if the trigger was a variation in the function that was not exercised during development but is now exercised in the field, it would be identified as a test variation trigger, belonging in the class of component or function test triggers. This enables us to make the assertion that this fault was really an escape of component function test and is, we believe, a much more objective assessment of the escape. Thus, for analysis such as we showed in Figure 2 which provides strong evidence of the different potential areas of escape we believe that identifying the trigger first and then attributing a potential escape is a superior mechanism to follow.