Applying Statistical Process Control to Automate the Analysis of Performance Load Tests

Abstract

Avoiding performance regressions is very important in the evolution of ultra-large software systems. Even the addition of an extra field or control statement can degrade the performance of the software system considerably as the impact of such simple changes is multiplied by the millions of user requests that are processed simultaneously. Performance testers conduct performance load tests and analyze the results of such tests, before every new deployment, to ensure that there are no performance regressions. However, such an analysis is challenging because each test run produces a large amount of performance counter (e.g., CPU and memory readings) data.

In this thesis, we propose approaches based on a statistical process control technique, called control charts, to automatically identify and determine the causes of performance regressions. We conduct case studies on three different software systems to evaluate our proposed approaches. The results show that our approaches can determine if a new test run has performance regressions. Compared to existing approaches, our approaches can detect performance regressions accurately in most of our case studies.