ODC - Orthogonal Defect Classification

Measurement of Failure Rate in Widely Distributed Software

Ram Chillarege, Shriram Biyani, and Jeanette Rosenthal

Center for Software Engineering
IBM TJ Watson Research Center, Yorktown Heights, NY
(914) 784 7375 ramchill@watson.ibm.com

Abstract:

In the history of empirical failure rate measurement, one problem that continues to plague researchers and practitioners is that of measuring the customer perceived failure rate of commercial software. Unfortunately, even order of magnitude measures of failure rate are not truly available for commercial software which is widely distributed. Given repeated reports on the criticality of software, and its significance, the industry flounders for some real baselines.

• This paper reports the failure rate of a several million line of code commercial software product distributed to hundreds of thousands of customers. To first order of approximation, the MTBF plateaus at around 4 years and 2 years for successive releases of the software. The changes in the failure rate as a function of severity, release and time are also provided.
• The measurement technique develops a direct link between failures and faults, providing an opportunity to study and describe the failure process. Two metrics, the fault weight, corresponding to the number of failures due to a fault and failure window, measuring the length of time between the first and last fault, are defined and characterized.
• The two metrics are found to be higher for higher severity faults, consistently across all severities and releases. At the same time the window to weight ratio, is invariant by severity. The fault weight and failure window are natural measures and are intuitive about the failure process. The fault weight measures the impact of a fault on the overall failure rate and the failure window the dispersion of that impact over time. These two do provide a new forum for discussion and opportunity to gain greater understanding of the processes involved.

• Introduction
• Service Process
  • Data
  • Failure/Fault Relationship
• Results
  • Fault Weight and Failure Window
• Summary
• References
• About this document ...