A probabilistic alternative to regression suites

Automated regression suites are essential in developing large applications, while maintaining reasonable quality and timetables. The main argument against the automation of regression suites, in addition to the cost of creation and maintenance, is the observation that if you run the same test many times, it becomes increasingly less likely to find bugs. To alleviate such problems, a new regression suite practice, using random test generators to create regression suites on-the-fly, is becoming more common. In this practice, instead of maintaining tests, we generate test suites on-the-fly by choosing several specifications and generating a number of tests from each specification. We describe techniques for optimizing random generated test suites. We first show how the set cover greedy algorithms, commonly used for selecting tests for regression suites, may be adapted to selecting specifications for randomly generated regression suites. We then introduce a new class of greedy algorithms, referred to as future-aware greedy algorithms. The algorithms are computationally efficient and generate more effective regression suites.