This repository is a companion to the paper "A Multi-Perspective Evaluation of Static Mutant Selection Techniques". It contains scripts to reproduce the results of the experiments. The data needed to reproduce the results can be found here. To reproduce download the figshare zip file and extract it at the root of the project.

To generate mutants using MediumDarwin tool, install the MediumDarwin tool:

• git clone https://github.com/ComplexSoftwareLab/MediumDarwin.git
• cd MediumDarwin
• pip install -r requirement.txt

To run MediumDarwin for a maven project, use:

• python MediumDarwin.py -m -b
  -p [path to production code (usually in src/main)]
  -t [path to build directory (usually the one containing pom.xml)]
  -c [build command separated by commas (usually mvn, test-compile, test)]
  -s [enable subsumption analysis output]
  --timeout (seconds) [timeout for mutant execution (default=120)]
  --all [use all operators]

Example: python MediumDarwin.py -m -b -p /../src/main -t /../ -c mvn,test-compile --test-command mvn,test -s --timeout=120 --all

Features of the different projects used in the experiments

Contains full information on all mutant selections produced in the experiments before aggregation in the format :

Where:

• Project: the project the mutants were generated from
• Technique: the mutant selection technique used to generate the selection
• Iteration: the iteration number, from 1 to number of repetitions (1000)
• Percentage: the sampling rate used used to generate the selection (or corresponding for OG)
• Rank: the rank of this technique at this sampling rate in this iteration

Contains agggregated information on all mutant selections produced in the experiments and their performance in the format :

Where:

• technique: the mutant selection technique used
• project: the project the mutants were generated from
• percentage: the sampling rate used for the technique
• min_Measure: the minimum RE/MMC/STC/MTS achieved by the technique on this project at this sampling rate (across 1000 runs for techniques involving randomness)
• mean_Measure: the mean RE/MMC/STC/MTS achieved by the technique on this project at this sampling rate (across 1000 runs for techniques involving randomness)
• max_Measure: the maximum RE/MMC/STC/MTS achieved by the technique on this project at this sampling rate (across 1000 runs for techniques involving randomness)
• std_Measure: the standard deviation of the RE/MMC/STC/MTS achieved by the technique on this project at this sampling rate (across 1000 runs for techniques involving randomness)
• median_Measure: the median of the RE/MMC/STC/MTS achieved by the technique on this project at this sampling rate (across 1000 runs for techniques involving randomness)

A zip file containing the data necessary to the reproduction of our results

Folder containing the output produced by MediumDarwin on the 14 projects used in the experiments

Folder containing the mutant sets produced by the different techniques on each project

Folder containing the mutant sets showing which mutants were "kiiled" or "survived"

Folder containing the output produced by the metric MMC on the 14 projects used in the experiments

Folder containing the output produced by the metric STC on the 14 projects used in the experiments

Folder containing the output produced by the metric MTS on the 14 projects used in the experiments

Folder containing the mutant-test matrix on each project, showing which tests kill each mutant

The scripts used in our experiments that can be used to replicate our results, they rely on the data available in Mutation_2026.zip and are located in the replication_script folder of this repository

Produces subsets of mutants using the hybrid selection techniques from mutant_status.csv's output and saves them as selection/technique_name/selected_mutants_project_name.csv

Produces subsets of mutants using the random selection sampling technique from mutant_status.csv's output and saves them as selection/technique_name/selected_mutants_project_name.csv

Produces subsets of mutants using the stratified selection techniques from mutant_status.csv's output and saves them as selection/technique_name/selected_mutants_project_name.csv

Produces subsets of mutants using the operator group techniques (selective mutation) from mutant_status.csv's output and saves them as selection/technique_name/selected_mutants_project_name.csv

Produces aggregated_results.csv for each of the metrics from the previous scripts' outputs and saves it in aggregated_results.csv

Visualises the data contained in full_results.csv for each of the metrics