ODC - Orthogonal Defect Classification

2. Defect Data

Defect data used for this paper comes from the development of operating systems code. There were very small portions of the code that needed to be in assembler, and for the most part the code was written in a high level language (e.g. C). The number of programmers and testers throughout the project ranged in the hundreds. The code included both new and changed code, given that there was an earlier base to work with. The defect data used for this paper included defects from functional test, system test, and early ship. By functional test we mean tests on specific modules and components that do not require the entire system to be up. System test required almost all the code to be ready. The defects found during these phases of testing are called Program Trouble Memorandums or PTM for short. The PTM database tracked defects from function test, system test and also for a short time after the product was released. Field defects, in IBM are commonly tracked in another data base called RETAIN. However, defects found during early ship are tracked by the developer as PTMs. This data, is thus what is tracked by the developer and includes some defects found after system test, during early ship. The PTM data is maintained in a database with a considerable amount of tracking information. Each defect entry contains information on: the date it was opened, it's current status, the date it was closed, the symptom of the error, a description, ownership, the person tracking it, etc. This is on on-line database and data entry is through panels with capability for searches. The specific fields in the defect entry contain tracking information which are also useful as search arguments on the database. Such fields help the generation of quantitative information such as reliability growth curves, distribution of severity, average time a defect is open, etc. However, the descriptive information recorded against a defect does not provide such conveniences. Yet, the descriptive information contains the meat, such as the problems experienced, tests used, and the fixes necessary etc. Thus, analysis that depends on the descriptive information in a defect is not easy to perform. Keyword searches are possible, however, they only help in a limited fashion. Such analysis invariably requires reading and comprehending a few paragraphs of descriptive text per defect. To aid the identification and tracking of defects, a defect entry is opened with a key for the symptom experienced by the failure. A symptom is what the tester notices as the effect of the defect on the system. The symptom may be the response to a test case, or a workload running on the system, or just commands issued by the tester. Symptoms are useful to testers who search the database for other defects with the same symptom to check whether the same problem was reported earlier within the same component. However, note that the symptom does not allude to any possible cause of the defect and only identifies its effects. A set of symptom descriptors such as: hang, data-lost, performance, loop, etc., are agreed upon to be used against defects. Not always is it possible to map a defect's symptoms into a very specific descriptor and therefore some descriptors have a more general scope, such as functional claim. Table 1 shows the set of symptom descriptors used and what they stand for. Since the data is keyed by symptoms, it provides a mechanism for us to partition the database. Another key used when the defect is opened is severity. This is a number between 1 and 4 and reflects a combination of the difficulty caused by the defects as also the urgency for a fix.

Symptom         Description S1   Defect resulted in data being lost S2   Hang; machine does not respond S3   Incorrect data was shown on display S4   Incorrect message displayed S5   Incorrect data output S6   Problem during build; Library S7   Performance problem (noticeable) S8   Unpredictable results occur S9   Useability of machine affected S10  Problems uncovered during a build S11  Unverifiable claim (no test case) S12  Functional claim (needs test case) S13  Performance claim (needs test case) S14  Architectural claim S15  Core dump (memory fault, abort)   Table 1.  The 15 Symptom Descriptors.