ODC - Orthogonal Defect Classification

Specific Issues

  
Figure 3: Review, Inspection Trigger Distribution

  
Figure 4: Function Test Trigger Distribution

  
Figure 5: System Test Trigger Distribution

In this subsection, we will look at the trigger profiles in Figures 3, 4 and 5 and discuss the trends we notice.

First, let us look at Figure 3 which shows the inspection class of triggers. These triggers are usually issues that can be attributed to design. The relatively flat distribution of 'design non-conformance' triggers over the first year and a half suggests that the customers are continuously poking and prodding the functional aspects of the product. Thus, relative to the other triggers, there is no sharp peak. Documentation and backward compatibility failures, if they occur, are likely to be uncovered very quickly. In contrast, lateral compatibility failures don't peak until almost a year later. The implication is that the product is able to handle environments which were available when it was introduced, but as the customer begins to install new products, which probably were not available when the product was being tested, faults begin to surface.

Second, let us focus on the 'function test' triggers as shown in Figure 4. Customers who are exploiting the product based on some knowledge of internals, tend to do so in the first year. This is inferred because both simple path and complex path triggers have little or no activity in the second year, and are mostly concentrated in the first year. However, note that between the two, simple path hits early, followed by complex path after two quarters.

Both test coverage and test variation triggers show a long, relatively flat distribution over the two year period. The probability of a test coverage fault surfacing is much higher in the first year, while test variation faults are much more likely to surface in the second year. The specific reasons for this phenomenon may be related to complexity, new software, new hardware, or the fact that the relatively simple problems are uncovered first before the customer becomes more adventurous. This is potentially an interesting avenue to explore, but is beyond the scope of this paper.

  
Figure 6: Fault Rate by Quarter

Finally, the system test trigger distributions, are shown in Figure 5. This shows perhaps the strongest story in terms of when one might expect specific faults to surface. The fact that hardware configuration failures are most likely to occur well into the second year, suggests that the bulk of hardware upgrades occur in that time frame. In contrast, software products appear to be upgraded very quickly, and frequently. The evidence that workload/stress failures do not show up until the second year indicates that customers probably don't push their system to it's limits until then. Startup/restart failures, if they occur, are likely to be uncovered immediately. The product's ability to recover gracefully from errors will be taxed throughout it's life.

In addition to the implications derived from examining when various triggers cause faults to surface in the field, another significant and unique aspect of a product relates to the volumes of each defect trigger reported by customers. These volumes appear to be influenced by both customer usage of the product and, more significantly, the relative success of fault removal and prevention during the development cycle relative to individual triggers.

Figure 6 shows the distribution over time of the faults associated with each of the three verification activities, by quarter. These are shown both by the volume of faults and the percent contribution during each quarter, to help look at the trends and the changes in the mix. The fact that the volume of review and inspect escapes and function test escapes far exceeds the volume of system test escapes has already been pointed out. It is interesting to note that, for this product at least, the distribution over time of the volume of system test escapes is relatively flat over time, while the volume of review and inspect escapes peak at about a year and then decreases rapidly. Thus, as time progresses, the percent of faults that are found primarily due to system test triggers increases, as should be expected. This view of the data enables us to form a prediction of field activity over time, while the views reflected in Figures 3, 4, and 5 offer a tremendous opportunity to target pre-release activities to effectively prevent or remove characteristic faults. The next progression is to quantify the expected benefit of actions identified to target specific triggers, evaluate the actual results in process, and adjust the field projection to accommodate these factors. Further expansion of this concept is, again, beyond the scope of this paper, but offers exciting potential for future investigation.