Effective test case selection for context-aware applications based on mutation testing and adequacy testing from a context diversityperspective

Abstract

Mutation testing and adequacy testing are two major technologies to assure the quality of software. In this thesis, we present the first work that alleviates the high cost of mutation testing and ineffectiveness of adequacy testing for context-aware applications. We also present large-scale multi-subject case studies to evaluate how our work successfully alleviates these problems. Mutation testing incurs a high execution cost if randomly selected test inputs kill a small percentage of remaining live mutants. To address this problem, we formulate the notion of context diversity to measure the context changes inherent in test inputs, and propose three context-aware strategies in the selection of test inputs. The empirical results show that the use of test inputs with higher context diversity can significantly benefit mutation testing in terms of resulting in fewer test runs, fewer test case trials, and smaller resultant test suites that achieve a high mutation score level. The case study also shows that at the test case level, the context diversity of test inputs positively and strongly correlates with multiple types of adequacy metrics, which provide a foundation on why context diversity contributes to the effectiveness of test cases in revealing faults in context-aware applications. In adequacy testing, many strategies randomly select test cases to construct adequate test suites with respect to program-based adequacy criteria. They usually exclude redundant test cases that are unable to improve the coverage of the test requirements of an adequacy criterion achieved by constructing test suites. These strategies have not explored in the diversity in test inputs to improve the test effectiveness of test suites. To address this problem, we propose three context-aware refined strategies to check whether redundant test cases can replace previously selected test cases to achieve the same coverage level but with different context diversity levels. The empirical study shows that context diversity can be significantly injected into adequate test suites, and favoring test cases with higher context diversity can significantly improve the fault detection rates of adequate test suites for testing context-aware applications. In conclusion, this thesis makes the significant contributions to the research in testing context-aware applications: (1) It has formulated context diversity, a novel metric to measure context changes inherent in test inputs. (2) It has proposed three context-aware strategies to select test cases with different levels of context diversity. Compared with the baseline strategy, the strategy CAS-H that uses test cases with higher context diversity can significantly reduce the cost of mutation testing over context-aware applications in terms of less number of test runs, smaller adequate test suites, and less number of test inputs used to construct test suites. (3) It has defined three context-aware refined strategies to construct adequate test suites with different context diversity levels. Compared with the baseline strategy, the strategy CARS-H that favors test cases with higher context diversity can significantly improve the effectiveness of adequacy testing in terms of higher fault detection rates.