ODC - Orthogonal Defect Classification

Deploying ODC at IBM

The deployment of a technology, such as ODC requires careful thought and considerable insight into the means of process insertion. Most practitioners would recognize that technology transfer is a difficult business. Process transfer is yet another order of magnitude harder in software. This is because, unlike a technology that can be captured in a tool or the design of a product, process transfer in software requires that every programmer change a little. Although the change may be very minor in terms of the actual work that a programmer has to do, getting acceptance of the concepts and the regular practice of it requires one to buy-in.

Providing the right education, which is not too lengthy and yet substantial so that the errors in classification are minimized is only the start. Unless programmers see the value of ODC quickly, it is hard to sustain the interest and commitment to provide good data. This quickly becomes an exercise in management of handling all the processes to execute ODC. Knowing the processes and having the right skill necessary is at best the minimum necessity. Being able to quickly recognize when they are not working and react to them effectively is the difference between success and failure. In our experience, we at Watson Research started as technologists, and teamed with our Divisional partners to do deployment. It quickly become evident that the split was artificial. We both needed to understand the technology an also the nuisances of the real world. To be effective, we learned what processes were necessary and developed schemes to maintain and police them.

Figure 5 identifies some of the key processes necessary. It also divides the range and scope of deployment into pilot, staged production, and production, indicating the growth of deployment in an IBM lab. The idea is that initially an organization would usually start off with a pilot project, as a trial. These usually last between three months to one year, and become the proving ground for ODC. Most of the processes were owned by Watson Research and the responsibilities of the participating lab were limited to classification and driving the actions that results from identifying in-process problems. As we made progress, we would develop the skills in the organization to own more of the processes, reducing the responsibility and involvement of the research team. In the best cases, we were able to get more than seventy percent of the processes owned at the labs, within eighteen months. At the end of 1993, we had close to 70 projects, in a dozen labs, with two of them seriously in the staged production phase. We expect it to be at least another couple of years before these labs completely own the processes.

The cost of doing ODC is quite low. There is an upfront cost in terms of modifying the tool set and providing education. The execution cost is dramatically low if a developer is already using a change control system. This is because, most change control systems, require that the programmers update a panel and the current ODC requires only 4 additional fields (in most cases). This is an incrementable cost that is negligible. The subsequent costs of data analysis are mostly tool costs and reviews with management or technical people of about 2-4 hours every couple of months. This can be rolled into existing quality programs or quality circle efforts, thereby not requiring an additional effort.

In cases where there is no change control tools used, nor any effort to look at software metrics, the cost is more substantial. The true cost is not so much in introducing ODC, but in developing a culture to look at metrics, data, and recognize the need and benefits of quality improvement.

  
Figure 5: Deployment of ODC

When ODC is used to enhance the Quality Circle or the Defect Prevention Process (DPP) [MJHS90] significant savings can be accrued in analysis costs. Typically, DPP like efforts cost in the range of one person-hour per defect. Imagine four people in a room analyzing defects. They usually do a detailed root cause analysis of around four or five defects in an hour. The one hour cost usually includes not only qualitative analysis, but also identifying a potential solution and writing it down as an action item for the organization to execute. Given such high costs, it is not uncommon for organizations not to be able to do DPP on every defect, since they probably have thousands of defects. The ODC classification, which extracts cause and effect usually takes only two minutes. Granted that the granularity of the measurement is much coarser, its low cost allows full coverage over the defect population. The analysis of these data, provide a statistical means to do causal analysis by associating cause and effect. This occurs, now, not on each defect, but on a collection of them, and is appropriately timed at the exit of a development phase. Since the analysis of the data (which may even be qualitative), is amortized over several of them, the overall cost is reduced. Our estimates show an order of magnitude, savings in analysis and decision making costs over traditional root cause analysis.