Abstract

There is an abundance of metrics and models in the field of software reliability engineering, but a significant challenge in their use is to develop a framework that is both mathematically sound and also reflects the behavior of software in a large, productional software environment. Most importantly, the metrics and models need to help the engineering teams to improve their practices and processes, and thereby enable them to produce great products.

High performance models are needed to enable software practitioners to identify deficient (and superior) development and test practices. Even using standard practices and metrics, software development teams can, and do, vary substantially in practice adoption and effectiveness. One challenge for researchers and analysts in these organizations is to develop and implement mathematical models that adequately characterize the health of individual practices (such as code review, unit test, static analysis, function testing, etc.). These models can enable process and quality assurance groups to assist engineering teams in surgically repairing broken practices or replacing them with more effective and efficient ones.

In this tutorial, we will describe our experience with model building and implementation, and describe the boundaries within which certain types of models perform well. We will also address how to balance model generalizability and specificity in order to integrate computational methods into everyday engineering workflow.

An important part of this modeling and analysis effort needs to be the correlative ‘linking’ of the development and test metric values to customer experience outcomes and then to customer sentiment (satisfaction). These linkages are essential in not only convincing engineering leadership to use computational tools in practice, but also in enabling investigators, at an early stage, to design experiments and pilots to test model applicability moving forward in time. After convincing experiments and pilots have been demonstrated, much work remains: Choosing a useful, and manageable, set of metrics, establishing goals and tracking/reporting mechanisms, and planning and implementing the tooling, training, and rollout. These practical considerations invariably put a strain on the models, therefore the models and ancillary analyses must be resilient, ‘industrial strength.’

Understanding a model’s practical limitations and strengths is an important part of its use – just as the mathematical and statistical limitations and strengths underscore a model’s scientific validity. Both factors, mathematical and practical, need to mesh properly in a workable way in a computation-driven engineering environment. The tutorial addresses the integration of these factors.

We will describe our experience in building and implementing models that are used by engineering teams that employ diverse development approaches, including waterfall, hybrid and agile development. We will show how we link in-process measures (from development and test) to customer experience and customer satisfaction, which in turn correlate strongly to company revenue. We will discuss the steps involved in choosing the most valuable metrics, setting goals for these metrics, and using them to help in improving development and test practices and processes.

Tutorial Outline